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DEFINITIONS 

IDA  publishes  the  lollowing  documents  to  report  the  results  o!  Its  worlt. 

Reports 

Reports  are  the  most  authoritative  and  most  carefully  considered  products  IDA  publishes. 
They  normally  embody  results  of  major  projects  which  (a)  have  a  direct  beariof!  an 
decisions  affecting  major  programs,  (b)  address  Issues  of  significant  concern  to  the 
Executive  Branch,  the  Congress  and/or  the  public,  or  (c)  address  Issues  that  have 
significant  economic  Implications.  IDA  Reports  are  reviewed  by  outside  panels  of  experts 
to  ensure  their  high  quality  and  relevance  to  the  problems  studied,  and  they  are  relessad 
by  the  Presldonl  of  IDA. 


Group  Reports 

Group  Reports  record  the  findings  and  results  of  IDA  established  working  groups  ana 
panels  composed  of  senior  Individuals  addressing  major  Issues  which  otherwise  would  bs 
the  subject  ol  an  IDA  Report.  IDA  Group  Reports  are  reviewed  by  the  senior  inillvhi-.iri- 
responslble  lor  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  an<! 
relevance  to  the  problems  studied,  and  are  released  by  the  President  of  IDA. 
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Papers  | 

Papers,  also  au'horltatlve  and  carefully  considered  products  ol  IDA,  address  studloc  thal  « 
are  narrower  In  scope  than  those  covered  in  Reports.  IDA  Popart  ara  reviewed  is  ; 
that  they  meet  the  high  standards  expected  of  refereed  papers  In  professional  journals  or 
lormal  Agency  reports. 

Documents 

IDA  Documenls  are  used  lor  the  convenience  of  the  sponsors  or  the  analysts  (a)  to  racord 
substantive  work  dons  In  quick  reaction  studies,  (b)  to  record  the  proceeding'-,  of 
conlerances  and  meetings,  (c)  to  make  avaltuble  preliminary  and  tentative  rssuiis  of 
analyses,  (d)  to  record  data  developed  In  the  course  of  an  Investigation,  or  (e)  to  toward 
inlormation  that  Is  essentially  unanalyzed  and  unevaluated.  The  review  ol  IDA  Docummis 
Is  suited  to  their  content  and  intended  use. 


.  c 

The  work  reported  in  this  document  was  conducted  under  contract  MDA  903  69  C  DGu3  for  ^ 
the  Department  ol  Oelonse.  The  publication  of  Ibis  IDA  document  does  not  IndlDafs  | 
endorsement  by  the  Department  of  Defense,  nor  should  Ihe  contents  be  censirund  as  | 
reliecling  the  ofliclal  position  of  that  Agency.  I 
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1A  ABSTRACT  (Idtdnjm  200  vmd$) 

Congress  directed  DoD  to  review  the  costs  and  capabilities  associated  with  the  use  of  the  Navy's 
Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF)  to  satisfy  aircraft  electronic  combat 
ground  testing  requirements  in  lieu  of  the  Air  Fmce  developing  its  own  Electronic  Ounbat  Integrated 
Test  (ECIT)  capability.  This  paper  examines  various  ACETEF  and  ECTT  capability  options  that  could 
reasonably  satisfy  stated  Air  Force  and  Navy  needs  in  terms  of  costs,  effectiveness,  and  other 
qualitative  considerations.  Costs  include  investn^ts  for  facilities,  equipment,  and  software; 
deployment  costs  for  testing  at  other  than  the  pimary  flight  test  location;  and  opoating  costs  fm- 
running  the  facilities  over  a  20-ycai  period.  Effectiveness  is  defined  as  the  percentage  of  requimnents 
satisfled  by  the  various  facility  and  capability  alternatives.  Based  on  the  aiialysis,  the  Air  Force  ECIT 
proposal  is  the  prefnred  alternative  for  meeting  both  stated  testing  and  many  variations  in  those  needs. 
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Cost  Effectiveiiess.  Air  ConAat  Environment  Test  and  Evaluation  Fadlity  (ACETEF),  Etectronic 
Combat  Integrated  Test  (ECIT)  Fi^Uty.  Test  and  Evaluation  (TftE).  Military  Facilities 
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INSTITUTE  FOR  DEFENSE  ANALYSES 

Contract  MDA  903  89  C  0003 
Task  T-N7-559 


PREFACE 


This  paper  was  prepared  by  the  Institute  of  Defense  Analyses  (IDA)  for  the  Office 
of  the  Director,  Test  and  Evaluation,  under  contract  MDA  903  89  C  0003,  Task  Order 
T-N7-559,  issued  16  November  1987/30  August  1991,  and  amendments.  The  objective  of 
this  portion  of  the  task  was  to  identify  the  most  cost-effective  alternative  among  Navy  and 
Air  Force  capability  options  that  satisfy  electronic  combat  ground  testing  requiremoits. 

This  work  was  reviewed  within  IDA  by  Robert  Dighton,  Thomas  Frazier, 
Stanley  Horowitz,  and  Alfred  Victor. 


EXECUTIVE  SUMMARY 


SUMMARY  OF  RESULTS 

The  Navy  and  Air  Force  estimate  their  requirements  for  electronic  combat/avicmics 
testing  at  installed  systems  test  facilities  (ISTFs)  to  be  about  7  shifts  per  day  beginning  in 
the  mid-1990s.  These  requirements  can  only  be  satisfied  by  expanding  current  ISTF 
aq>acity.  Such  expansion  must  include  a  large  anechcnc  chamber  (LAC)  because  about  half 
of  the  7-shift  estimate  is  dedicated  to  testing  of  large  aircraft 

Proposed  expansions  to  the  existing  Air  Cmnbat  Envinmmental  Test  and  Evaluatitm 
Facility  (ACETEF)  would  satisfy  only  36%  of  Service  requirements  and  none  of  the  large 
chamber  requirements.  Cmisequently,  we  considered  duee  additional  alternatives: 

•  Build  a  large  chamber  at  ACETEF 

•  Develop  an  Electronic  Combat  Integrated  Test  (ECFT)  capability  to  integrate 
with  the  existing  large  chamber  (Benefield  Anechoic  Facility  or  BAF)  at 
Edwards  Air  Force  Base 

•  Pursue  both  of  die  above  simultaneously. 

Pursuing  bodi  simultaneously  is  the  most  expensive  option  and  satisfies  idxHit  94% 
of  the  stated  requirements.  Another  strategy  would  be  to  select  one  of  the  other  two  less 
cosdy  options  and  defer  further  ISTF  expansion  until  additional  testing  experience  has  been 
gained. 

The  two  less  cosdy  alternatives,  budding  an  LAC  at  ACETEF  or  developing  the 
ECTT  capability  at  Edwards,  would  cost  about  the  same;  however,  develt^iing  an  ECIT 
capability  would  satisfy  mcxe  of  both  the  lequiiemmits  and  the  qualitative  considoadons. 
Therefore,  our  recommendation  is  to  develop  the  ECTT  ctqialnlity  now  and  retain  die  option 
to  add  an  LAC  at  ACETEF  if  future  demands  wairanL  To  ensure  the  ECTT  cost  benefits 
are  realized,  the  Air  Fence  should  commit  funding  to  complete  a  fuU  ISTF  in  time  to  test 
F-22  electronic  combat  systems  at  that  facility. 

The  subsections  that  follow  give  die  background  em  this  issue  and  explain  how  we 
arrived  at  the  idiove-stated  results. 


BACKGROUND 

Modem  aircraft  electronic  combat/avionics  systems  ate  becoming  increasingly 
ctHnplex  and  testing  of  such  systems  is  becoming  more  difficult  To  address  this  difficulty, 
the  Dquutment  of  Defoise  (DoD)  has  placed  ccmsiderable  emphasis  on  a  class  of  integrated 
ground  test  complexes  known  as  installed  systems  test  facilities  (ISTFs).  The  Navy 
operates  such  a  facility,  the  Air  Combat  Environmental  Test  and  Evaluation  Facility 
(ACETEF),  at  Patuxent  River  Naval  Air  Station  (NAS),  Maryland. 

The  DoD  FY  1992  budget  request  to  Congress  included  $10  million  for  the  Air 
Force  (AF)  to  begin  development  of  the  Electronic  Combat  Integrated  Test  (ECIT)  facility 
at  the  AF  Flight  Test  Center,  Edwards  Air  Force  Base  (AFB),  California.  The  Senate 
Apixx>priations  Committee  deferred  action  on  the  request,  eliminated  all  1992  funding,  and 
noted  that  the  total  ECTT  upgrade  would  cost  $250  million.  Also,  the  Committee  stated  that 
ECIT  was  not  justified  because  of  "the  possibility  of  substituting  a  less  costly  modificadtxi 
to  a  Navy  facility  and  because  firm  test  requirements  for  ECIT  are  lacking”  [1].  The 
Committee  directed  DoD  to  compare  the  cost  of  modifying  the  Navy's  ACETEF  with  the 
cost  of  upgrading  ECIT.  The  results  were  to  be  submitted  with  the  FY  1993  budget 
request 

Subsequently,  the  House  and  Senate  Appropriations  Conference  Committee's 
report  of  18  November  1991  also  eliminated  the  $10  million  funding  request  and  directed 
that  no  other  funds  be  used  to  support  the  ECIT  upgrade.  The  Conference  Committee 
asked  that  DoD  "review  the  capabilities  of  the  Navy’s  Air  Combat  Environment  Test  and 
Evaluation  Facility  to  determine  the  feasibility  of  using  this  facility  for  Air  Force  testing 
lequiiements"  and  {xovide  the  results  to  the  House  and  Senate  ^)pfDpriations  Committees. 

In  response  to  the  congressional  request,  the  Director,  Test  and  Evaluation,  Under 
Secretary  of  Defense  (Acquisition),  asked  the  Institute  for  Defmise  Analyses  (IDA)  to 
assess  the  costs  associated  with  ACETEF  and  ECTT  alternatives  and  the  ctq)abilities  they 
represent  with  respect  to  Air  Force  and  Navy  requirements.  IDA  was  further  asked  to 
make  use  of  Service  estimates  to  the  maximum  extent  possible.  The  results  of  our 
subsequent  analysis  depended  on  three  parameters;  the  requirements  for  electtxHiic 
combat/avionics  installed  systems  test  facilities,  the  costs  of  various  combinatimis  of  ISTFs 
at  Patuxmit  River  NAS  and  Edwards  AFB,  and  the  resulting  capabilities  to  meet  the  Service 
requiremaits. 
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OVERVIEW  OF  ANALYSIS 
Test  Requirements 

First,  we  reviewed  future  Navy  and  Air  Force  aircraft  program  requirements  for 
installed  systems  test  facilities.  Table  ES-1  sununatizes  those  requirements.  The  combined 
Navy  and  Air  Force  test  requirements  are  equivalent  to  almost  7  shifts  of  anechoic  chamber 
use  per  day.  About  half  of  the  requirements  are  for  aircraft  that  require  a  large  anechoic 
chamber  (LAC). 


Tabis  ES-1.  Future  Raquirumants  for  ISTF  Chambors 
(Avaraga  Shifta  Par  Day) 


Usa 

Small 

Anecboic  Chamber 

Large* 

Anecboic  Chamber 

Total 

Fbvy 

2.0 

1.8 

3.8 

Air  Faroe 

1.4 

1.7 

3.1 

Toud 

3.4 

3.5 

6.9 

*  *L«rga*  raqniP-iiMiitt  m  dwie  that  cannot  be  aatianed  in  the  exiating 
ACEIEF  chamber. 


We  judged  diese  requiremrats  to  be  reastmable  because: 

(1)  They  were  validated  by  program  manages  (PMs),  who  are  not  motivated  to 
overstate  test  requirements. 

(2)  They  were  consistent  with  the  trends  of  increasing  i»opoilions  of  flyaway 
costs  attributable  to  electronic  combat/avionics  systems.  Such  systems 
incorporate  ccnnplex  technology  and  represent  developmental  and  operational 
challrages. 

(3)  They  were  consistent  with  Navy  experience  at  ACETEF.  The  Navy  uses  the 
ACETEF  chambo'  an  avoage  of  U  shifts  per  day,  S  days  per  wedt,  for  snoall 
aircraft  now,  and  expects  usage  to  increase  to  an  average  of  2  shifts  per  day  by 
1994  and  beytmd.  The  Navy  estimates  its  requiremoits  for  large  chamber 
testing,  fcM’  uiuch  there  is  no  current  capability,  at  1.8  shifts  per  day.  This 
estimate  is  ctmsistent  with  the  Navy's  small  chambo*  experience.  Air  Force 
requirements  are  comparable  to  those  of  the  Navy,  both  in  total  and  by 
individual  aircraft,  and  like  the  Navy's  requirements,  may  increase  as 
experience  is  gained  widi  die  use  of  ACElEF-type  facilities. 

(4)  Ground  testing  is  cost-effective.  While  not  a  complete  substitute  for  flight 
testing,  ground  testing  can  accomplish  many  test  objectives  at  about  12%  of  the 
cost  of  poforming  similar  objectives  in  flight  testing.  Moreover,  ground 
testing  can  synthesize  high-density  electronic  signal  environments,  which 
cannot  be  created  for  flight  testing. 

To  further  test  their  reasonableness,  we  conducted  smisitivity  analyses  at  50%, 
75%,  125%,  and  150%  of  the  stated  requiremoits. 
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Alternatives  To  Satisfy  Requirements 

We  considered  four  alternatives  that  satisfy  varying  levels  of  the  total  test 
requirements.  Each  alternative  includes  the  cost  and  capabilities  of  currently  programmed 
enhancements  to  ACETEF  contained  in  the  Central  Test  and  Evaluation  Investment 
Program  (CTEIP).  In  order  of  increasing  cost  and  ability  to  meet  requirements  the 
alternatives  are  as  follows: 

•  Alternative  1:  Operate  ACETEF  at  2.5  shifts  per  day  instead  of  the  current  1.5. 
C^timal  schedu^g  would  raise  the  pofcentage  of  the  requirements  that  could 
be  met  from  22%  to  36%.  This  alternative  would  not  provide  for  a  large 
anechoic  chamber,  and  almost  half  of  the  services'  requirements  are  for  testing 
in  a  large  anechoic  chamber. 

•  Alternative  2:  Build  a  large  anechoic  chamber  at  Patuxent  River  NAS  and 
connect  it  to  the  ACETEF  laboratories.  With  optimal  scheduling,  ACETEF 
could  support  2  shifts  in  each  chunb^  for  a  total  of  4.  This  alternative  would 
raise  the  percentage  of  requirements  met  from  22%  to  57%. 

•  Alternative  3:  Develop  the  EOT  to  serve  the  large  chamber  at  Edwards  AFB 
(Benefield  AiMchoic  Facility  or  BAF)  and  operate  ACETEF  at  2.5  shifts  per 
day.  ECTT  would  also  operate  at  2.5  shifts  per  day.  This  alternative  would 
accommodate  72%  of  the  stated  requirements  instead  of  the  current  22%. 

•  Alternative  4:  Build  the  LAC  at  ACETEF  and  develop  the  EOT  at  Edwards, 
combining  Alternatives  2  and  3.  This  altomative  would  accommodate  94%  of 
the  stated  Navy  and  AF  requirements. 


Hgure  ES-1  shows  grsqihically  the  amount  of  total  test  requirements  each  of  the  altmoadves 
could  satisfy  as  compared  to  the  baseline. 
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Figur*  E8-1.  Reqidrwnwits  Satisfied  by  the  AKamatlvas 
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Costs  and  Effectiveness  of  Alternatives 

In  estimating  the  costs  associated  with  the  four  alternatives,  we  considered  only 
those  costs  that  differed  among  the  alternatives.  Costs  such  as  those  incurred  by  actual 
testing  were  not  included  when  they  were  assumed  to  be  the  same  at  one  location  as 
another.  Major  cost  elements  considered  were  investment,  20  years  of  operations,  and 
deployment  Only  the  last  category's  cost  estimate  had  significant  uncertainty.  The  F-22 
Systems  Program  Office  (SPO)  estimated  that  it  would  cost  $309  million  more  in  constant 
FY  1992  dollars  to  deploy  to  ACETEF  that  portion  of  the  F-22's  EC/avionics  ground  test 
program  involving  an  ISTF  than  to  conduct  that  ground  testing  at  Edwards,  where  flight 
testing  and  other  ground  testing  will  be  conducted.  This  estimate  included  the  cost  to 
construct  a  systems  integration  laboratory  (SIL)  with  two  avionics  bench-test  lines  at 
Patuxent  River  NAS  adjacent  to  ACETEF  and  its  chamber. 

The  AF  rationale  for  this  estimate  was  that  the  F-22  is  a  major  new  development 
that  incorporates  the  most  technologically  advanced  (integrated)  electronic  combat/avionics 
systems  ever  undertaken  by  the  DoD.  As  a  result,  the  Air  Force  considered  it  essential  that 
a  SIL  (or  hardware-in-the-loop)  facility  be  collocated  with  the  ISTF  if  the  ISTF  is  not 
collocated  with  the  flight  test  program.  The  $309  million  represents  the  total  cost  to  the  F- 
22  program  associated  with  build-up  and  26  months  with  an  F-22  avionics  test  aircraft  in 
an  anechoic  chamber  at  ACETEF.  At  the  other  end  of  the  spectrum,  similar  deployment  to 
ACETEF  of  previous  operational  aircraft  for  tests  of  limited  duration  and  scope  without  a 
collocated  SIL  was  estimated  to  cost  only  $15  million. 

IDA  developed  a  "middle  ground"  deployment  estimate  of  $140  million,  which 
included  building  a  single-line  SIL  at  ACETEF  with  fewer  people,  fewer  spares,  less 
equipment,  and  less  duplicative  flight  testing  than  included  in  the  F-22  estimate.  We 
believed  this  estimate  to  be  a  reasonable  balance  between  risk  and  affordability. 

Table  ES-2  shows  the  investment,  20  years  of  operation,  and  deployment  cost  of 
the  four  alternatives  using  the  IDA  deployment  estimate.  (All  three  deployment  estimates, 
$15  million,  $140  million,  and  $309  million,  are  covered  in  Section  VI  of  the  main  report 
and  in  the  requirements  excursions  in  Appendix  E.)  The  details  of  these  cost  estimates  are 
presented  in  Section  VI  and  in  the  appendices. 

Figure  ES-2  illustrates  the  discounted  costs  for  the  same  four  alternatives  shown  in 
Table  ES-2.  It  also  shows  the  difference  between  the  percent  of  test  requirements  (needs) 
met  by  each  alternative  and  the  22%  met  by  the  current  ACETEF  operation.  The  scale  at 
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the  left  of  the  figure  measures  the  costs  of  each  alternative,  and  the  scale  at  the  right 
measures  the  percentage  of  needs  met  Note  that  the  F-22  deployment  costs  from  Edwards 
AFB  to  Patuxent  in  Alternatives  1  and  2  offset  much  of  the  increased  investment  and 
operations  costs  for  Alternatives  3  and  4  that  include  developing  the  ECIT  facility  at 
Edwards. 


Table  ES-2.  Costa  of  AKarnativas 


Descriotioo  of  Alternative 

Millions  of 
Constant  FY 
1992  Dollars 

Millions  of 
Discounted  FY 
1992  Dollars 

1 

Provide  2.5  Shifts  at  ACbrTt^ 

$262 

$167 

2 

Build  LAC  at  Patuxent  with  4-Sbift  Csqiacity 

$454 

$249 

3 

Develop  ECTT  at  Edwards  (and  AIL  1  ACETEF) 

$486 

$264 

4 

BuUdLAC  and  EOT  (Alt  2  +  Alt  3) 

$589 

$328 

Note:  Costs  include  future  investments,  20  years  operations,  and  deploymenu,  using  IDA's 
estimate  for  F-22  deployment  Discounted  costs  are  at  10%  per  annum. 
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Figure  ES>2.  Comparison  of  AKarnativaa 


Findings 

TIm  three  general  findings  presented  below  are  supported  by  more  detailed  and 
specific  findings  in  Secticm  Vm  of  the  main  report 

•  There  is  a  demonstrated  i^ed  for  more  ISTF  capability  as  a  much  cheaper 
(compared  to  flight  testing)  and  a  more  effective  approach  for  cotain  kinds  of 
testing. 
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•  Although  using  the  improved  ACETEF  to  satisfy  Air  Force  requirements  takes 
advantage  of  superior  Navy  expertise,  it  adds  considerable  deployment  costs 
and  is  inconsistent  with  the  AF  philosophy  of  combining  ground  and  flight 
testing  at  one  locaticm. 

•  Although  the  Air  Force  plan  for  ECTT  is  incomplete  and  ECFTs  potential  may 
be  difficult  to  realize  within  a  short  period,  the  long-term  advantages  both  in 
terms  of  effectiv^iess  and  cost  outweigh  the  risk. 

OTHER  CONSIDERATIONS 

Costs  and  capabilities  were  the  major  considerations  in  our  analysis;  however, 
several  other  issues  should  be  considered  before  any  decision  is  made.  These  other 
considerations,  discussed  only  briefly  here,  are  covered  more  thoroughly  in  Sections  VU 
and  Vni  of  the  main  paper. 

Other  Studies.  Affordability  issues  aside,  our  review  of  the  test  requirements 
provided  by  the  Air  Force  and  the  Navy  supports  the  findings  of  several  prior  studies  that 
recommend  the  develofHnent  of  more  and  better  installed  systems  test  facilities. 

Project  Reliance:  The  Need  to  Make  It  Work.  Project  Reliance  is  an  initiative 
designed  to  promote  efficient  multi-Service  use  of  Test  and  Evaluation  (T&E)  investment 
and  facilities  and  to  eliminate  unwarranted  duplicatitm  of  T&E  facilities.  Recommendations 
of  a  Reliance  study  group's  assessment  of  test  support  resources  highlight  the  need  to 
upgrade  the  EC/avionics  ground  test  capabilities  at  both  Patuxent  and  Edwards.  The 
framework  is  now  in  place  to  begin  a  coordinated  multi-Service  ^proach  to  improving 
ISTFs  while  minimizing  potential  duplication.  This  process  should  be  given  the 
opportunity  to  work. 

Availability  of  Experienced  Work  Force.  Because  the  ACETEF  work  force  is 
already  experienced,  building  an  expanded  capability  at  Patuxent  River  would  be  easier  and 
quicker  than  starting  an  essentially  new  ctq)ability  at  Edwards. 

Navy  Expertise.  The  existing  capabilities  at  Patuxent  for  the  ground  testing  of 
electronic  combat  installed  systems  ate  second  to  none.  However,  even  if  a  new  large 
anechoic  chamber  were  built  at  Patuxent,  the  capacity  there  would  be  insufficient  to  meet  all 
of  tte  expected  current  and  future  electronic  combat  system  testing  needs. 

Service-Unique  Requimnents.  ACETEF  support  of  Navy  programs  may  suffer  if 
required  to  support  higher-priority  Air  Force  aircraft  programs  such  as  the  F-22  over  an 


extended  period.  The  problem  would  be  particularly  acute  if  the  Navy's  A-X  development 
program  accelerates  and  the  F-22  schedule  slips. 

Increased  Demand  for  Ground  Testing.  Experts  believe  that  future  demand  for 
ground  testing  of  EC/avionics  systems  will  increase  as  the  complexity  and  integration  of 
avionics  subsystems  increases  and  the  need  for  significant  ground  testing  grows.  The  AF 
has  a  wide  variety  of  aircraft  that  can  benerit  from  ISTF  testing. 

Potential  to  Meet  Workload  Surges.  Even  if  one  ISTF  had  the  capacity  to  do 
sufficient  work  to  satisfy  critical  DoD  requirements,  there  would  be  value  in  having  two 
facilities  to  meet  workload  surges  or  to  compensate  for  the  catastrophic  loss  of  one  facility. 

New  Defense  Acquisition  Strategy.  Deputy  Secretary  of  Defense  Atwood's  new 
approach  to  Defense  acquisition  emphasizes  technology  demonstration  and  prototype 
evaluation.  Such  a  policy  requires  a  strong  EC/avionics  test  capability. 

Relative  Costs  of  Test  Facilities  and  Weapon  Systems.  Some  dupUcation  of  test 
assets  may  be  appropriate  to  minimize  the  cost  of  weaptm  system  acquisitiotL  A  theoretical 
example  would  be  procuring  two  $10  million  test  instrumentation  packages  in  order  to 
prevent  production  delays  costing  $100  million.  A  relatively  small  test  investment  may 
leverage  large  acquisition  investments  and  significantly  reduce  overall  acquisition  costs. 

Program  Imnact.  Total  acquisition  cost,  not  test  facility  cost  alone,  should  be 
optimized.  The  cost  of  test  and  evaluation  (T&E)  facilities  is  not  a  major  driver  in 
determining  what  the  best  acquisition  strategy  should  be.  For  a  new  program  such  as  the 
F-22,  the  cost  of  an  interrupted  flight  test  program  at  Edwards  and  of  extensive 
deployments  to  Patuxent  risk  iiKreasing  system  acquisition  costs  substantially. 

Potential  for  Recovery  of  Test  Facility  Investment.  An  opportunity  exists  to 
recover  test  capability  investments  during  peacetime  acquisition  by  avoiding  test  and 
schedule  delays,  reducing  requirements  for  flight  testing,  and  preventing  aircraft  losses 
during  test  and  operations.  Costs  may  also  be  recovered  by  reducing  weapon  systems 
acquisition  expenses  (e.g.,  less  test  flight  hours,  reduced  procurement  quantities)  or  by 
avoiding  many  of  the  software  fixes  that  occur  during  production  and  operation. 

RESULTS  OF  THE  ANALYSIS 

Our  analysis  shows  that  the  current  ACETEF  with  an  additional  (intermediate  or 
large)  anechoic  chamber  and  an  upgraded  ECIT  facility  to  complement  the  LAC  at  Edwards 
would  satisfy  94%  of  future  requirements.  The  estimated  total  20-year  costs  for  providing 
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that  capability  is  $589  million  in  constant  FY  1992  dollars  ($328  million  in  dollars 
discounted  at  10%  per  annum).  Other,  less-costly  options  are  available.  One  of  these 
alternatives,  increasing  the  number  of  ACETEF  shifts  to  2.5  per  day,  would  satisfy  less 
than  two-fifths  of  the  stated  requirements  at  a  20-year  cost  of  about  half  that  of  building 
two  addititmal  chambers,  but  would  provide  no  large  aircraft  ISTF  c^ability. 

Of  the  other  two  alternatives,  building  a  large  anechoic  chamber  at  ACETEF  and 
increasing  the  number  of  supporting  shifts  to  four,  or  developing  an  upgraded  ECFT 
capability  at  Edwards  while  maintaining  the  2.5  shifts  at  ACETEF,  the  20-year  syst^ 
costs  are  apivoximately  the  same.  However,  developing  the  ECFT  capability  would  satisfy 
25%  more  requirements  than  building  an  LAC  at  ACETEF. 

The  increase  in  requirements  met  and  the  cost  of  that  increase  for  each  of  the 
alternatives  we  examined  is  shown  in  Table  ES-3.  The  last  colunm  shows  the  cost  per  unit 
of  additional  needs  met,  a  rough  measure  of  effectiveness  relative  to  cost  Note  that  the 
trend  of  this  cost  per  unit  of  added  effectiveness  is  precisely  opposite  that  of  total  additive 
cost  In  other  words,  effectiveness  is  gained  faster  in  successive  alternatives  than 
additional  costs  would  be  incurred.  The  cost  per  percent  of  added  needs  met  for  Altanadve 
2  is  almost  a  third  higher  than  for  Alternative  3. 


Table  ES-3.  Cost  Par  Additloiial  Raquiramant  Mat 


Descriution  of  Alternative 

Percent 
of  Needs 
Met 

taoease 
of  Needs 
Met 

AddUive 

Dbcouited 

Costs* 

Cost  per  Percent 
of  Added  Needs 
Met 

0 

Baseline  ACi^'iiiF 

22% 

0% 

$50 

NA 

1 

Provide  2.S  Sbifts  at  ACETEF 

14% 

$167 

$11.9 

2 

Build  LAC  at  Patuxent  with  4-Sliift  Capacity 

57% 

35% 

$249 

$7.1 

3 

Build  EOT  at  Edwards  (and  AIL  1  ACETEF) 

72% 

50% 

$264 

$5.3 

4 

Build  LAC  and  EOT  (Alt  2  -f  Alt  3) 

94% 

72% 

$328 

$4.6 

Note;  Costs  are  in  millions  of  FY  1992  doUvs  disoountod  at  10%  per  annian.  See  TaMe  ES-2  for  constant  dollars. 
*  Costs  include  future  investments,  20  years  of  operatimu,  and  dq|>loyments,  using  IDA's  estimate  for  F-22 


deployment  Discounted  costs  am  at  10%  per  annum. 


As  previously  observed,  fulfilling  most  of  the  stated  requiiments  would  require  a 
large  ACETEF  chamber  and  an  upgraded  ECIT  capability.  However,  given  the  declining 
defense  budget  and  tlw  intense  competition  for  limited  resources,  affordability  is  also  a 
major  consideration.  A  more  conservative  option  would  be  to  develop  an  upgraded  EOT 
now.  This  approach  would  satisfy  a  large  percentage  of  the  requiranents,  including  die 
new  capability  to  test  large  aircraft  Evoi  if  funds  were  not  fully  (and  pohaps  prematurely) 


committed  for  two  large  chambers  before  a  clear  historical  basis  or  commitment  had  been 
established,  this  alternative  would  still  provide  an  important  hedge  against  the  catastrophic 
loss  at  one  facility.  It  would  also  provide  the  greatest  flexibility  by  collocating  a  high- 
quality  installed  systems  test  capability  with  the  flight  testing  mission  at  Edwards.  It  also 
follows  the  recommendations  of  Project  Reliance  by  expanding  electronic  combat  ISTF 
capabilities  at  both  ACETEF  and  ECFT.  Requirements  should  continue  to  be  monitored 
over  the  next  several  years  to  determine  if  another,  larger,  anechoic  chamber  is  justified  at 
ACETEF. 

In  order  to  make  the  ECIT  alternative  competitive  in  our  analysis  above,  we 
assumed  that  the  Air  Force  was  willing  to  invest  an  estimated  $168  million  in  constant  FY 
1992  dollars  for  a  true  ISTF  operation  at  Edwards  AFB,  plus  additional  investments  unique 
to  the  F-22  (necessary  wherever  the  F-22  is  tested).  Furthermore,  development  of  a  fully 
functional  ECIT  caq)ability  would  have  to  be  planned  and  executed  quickly  if  it  is  to  be 
useful  for  the  F-22  development  testing. 

We  suggest  that  OSD  require  the  Air  Force  to  program  and  commit  to  such  a 
program  if  OSD  is  to  recommend  this  alternative  to  the  Congress.  Moreover,  signiEcant 
savings  should  accrue  if  the  Air  Force  uses  ACETEF-developed  technologies  and 
approaches  as  is  intended  with  the  funding  provided  by  OSD  in  its  CTEBP  funding  for 
ACETEF. 
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1.  INTRODUCTION 


The  Department  of  Defense  (DoD)  test  community  is  faced  with  a  declining  defense 
budget  and  iiKieased  competition  for  limited  resources.  At  the  same  time,  DoD  is  initiating 
new  acquisition  policies  and  complex  new  wetqwns  systons  that  are  expected  to  generate 
increased  test  requirements.  Eliminating  duplicative  facilities  and  consolidating  resources  is 
one  way  to  generate  budget  savings.  This  paper  examines  the  efficiency  and  effectiveness 
of  the  potential  consolidation  of  Navy  and  Air  Force  electronic  warfare  installed  systems 
test  facilities  fcxr  ground  testing  of  electronic  conbat/avionics  systems. 

A.  BACKGROUND 

Modem  electronic  combat/avionics  systems  for  aircraft  are  becoming  increasingly 
complex.  To  test  such  systems  comprehensively,  the  DoD  is  placing  increased  emphasis 
on  a  class  of  integrated  ground  test  complexes  known  as  installed  systems  test  facilities 
(ISTFs).  Consistent  with  diese  tnmds,  the  Dq)artment  of  Defense  FY  1992  budget  request 
to  Congress  included  $10  million  for  Che  Air  Force  (AF)  to  start  to  develop  an  Electronic 
Combat  Integrated  Test  OBCTT)  cq»bility  at  the  AF  Flight  Test  Center  at  Edwards  Air 
Force  Base  (AFB),  Califcnnia.  The  Soiate  .^rpropriations  Committee  deferred  action  on 
die  request  and  eliminated  all  1992  funding  [1],  noting  diat  die  expected  total  ECTT  upgrade 
would  cost  $250  millicm  and  stating  that  "ECTT  is  unjustified  due  to  the  possibility  of 
substituting  a  less  costly  modification  to  a  Navy  facility  and  because  firm  test  requiremoits 
for  ECrr  are  lacking."  The  Committee  further  directed  DoD  to  compare  the  cost  of 
modifying  the  Navy's  Air  Combat  Environmoit  Test  and  Evaluatitm  Facili^  (ACETEF)  at 
Patuxent  Rivor  Naval  Air  Station  (NAS),  Maryland,  with  the  cost  of  upgrading  ECTT.  The 
results  were  to  be  submitted  with  the  FY  1993  budget  request 

Subsequendy,  the  House  and  Senate  Approfuiation  Conference  Committee,  in  its 
report  of  18  November  1991,  also  eliminated  the  $10  million  funding  request  and  directed 
that  no  other  funds  be  used  to  support  the  ECIT  upgrade.  The  Confoence  Committee 
further  directed  that  DoD  "should  review  the  c^abilities  of  the  Navy's  Air  Combat 
Environmoit  Test  and  Evaluadtm  Facility  to  determine  the  feasibility  of  using  this  facility 
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for  Air  Force  testing  requirements.  Hie  Department  should  provide  this  review  to  the 
House  and  Senate  Appropriations  Committees." 

B.  SCOPE 

In  response  to  the  congressional  request,  the  Director,  Test  and  Evaluation,  asked 
the  Institute  for  Defense  Analyses  (IDA)  to  determine  the  costs  associated  with  die  various 
ACETEF  and  ECIT  alternatives  and  to  relate  those  costs  and  the  capabilities  they  refnesent 
to  Air  Force  and  Navy  requirements.  We  did  not  consider  other  locatitms  or  other  options, 
such  as  moving  ACETEF  to  Edwards  AFB  or  China  Lake  NAS.  IDA  was  further  tasked 
to  use  Service  estimates  to  the  maximum  extent  possible.  We  have  followed  that  tasking 
while  making  sensitivity  analyses  where  iqipropriate.  The  Hnal  result  is  highly  dependent 
upon  three  critical  parameters:  electronic  combat/avionics  test  requiremcmts,  the  costs  and 
effectiveness  achieved  by  various  combinations  of  installed  systems  test  facilities  at 
Patuxent  River  NAS  and  Edwards  AFB,  and  the  resulting  capabilities  to  meet  the  Service 
requirements. 

C.  OBJECTIVE 

The  primary  objective  of  diis  study  was  to  identify  the  best  ISTF  altnnative  among 
various  ACETEF  and  ECTT  opticms  that  might  satisfy  electronic  combat  ground  testing 
requirements  for  installed  systems.  This  task  includes  assessing  both  current  and  planned 
testing  requirements  and  curabilities  and  eAimatmg  die  related  costs.  The  paper  specifically 
addresses  costs  and  effectiveness  in  terms  of  available  c^iracity  and  projected  workload. 

D.  APPROACH 

Our  approach  is  based  on  three  interrelated  steps.  First,  we  relate  existing  and 
planned  ISTF  test  capabilities  to  future  Navy  and  Air  Force  workload  requirements. 
Second,  we  develop  alternative  packages  of  ISTF  capabilities  that  meet  various  levels  of 
test  requirements.  Third,  we  assess  the  relative  costs  and  effectiveness  of  each  of  the 
altemadves. 

The  Director,  Test  and  Evaluation,  requested  that  IDA  use  Service>developed 
electronic  combat  (EC)  test  requirements  and  costs  wherever  possible.  This  is  the  basis  of 
our  study,  given  the  complexity  and  difficulty  of  the  issues.  The  focus  of  our  approach 
was  on  reviewing  and  assessing  Service-provided  information  for  reasonableness, 
completeness,  and  consistency.  We  primarily  relied  on  Service  input  and  kept  our  own 
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original  estimates  to  a  minimum.  In  cases  where  our  assessment  of  risk  differs 
substantially  from  Service  estimates,  we  added  our  own  estimates  to  those  of  the  Service 
by  performing  sensitivity  analyses.  Our  approach  also  included  the  following  specific 
elements: 

•  We  obtained  written  information  from  the  Navy  and  Air  Force  regarding  test 
program  installed  systems  test  facility  requirements  dealing  with  EC,  test 
program  cost  estimates,  facility  EC  test  capability/cost  estimates,  and  facility 
upgrade  or  modificaticm  cost  estimates. 

•  We  reviewed  recent  studies  on  electronic  combat  testing  requirements, 
facilitks,  and  C2q>abilities. 

•  We  visited  the  ACETEF  and  EOT  facilities  and  discussed  Service-provided 
information.  We  received  additional  related  briefings  and  other  requested 
information  as  required. 

•  We  assessed  information,  conducted  cost  sensitivity  analyses  and  reviewed 
results  with  Service  points  of  ctmtact. 

E.  OUTLINE  OF  THE  REPORT 

In  this  paper,  we  present  the  marginal  (additional)  costs  and  c^abilities  estimates 
for  the  ACETEF  and  ECTT,  our  analysis  of  those  estimates  and  supplementary  analyses, 
and  our  findings  and  recommendations  resulting  from  those  analyses.  Section  II 
establishes  the  framework  for  analysis,  including  descriptions  of  the  test  requirements, 
alternatives  to  meet  those  requirements,  the  types  of  costs  incurred  with  those  alternatives, 
and  other  available  studies  and  considerations.  Section  m  provides  an  overview  of  the 
nature  of,  and  test  facilities  for,  testing  for  electronic  combat  ctqnbility.  This  overview  is 
intended  as  background  and  includes  descriptions  of  the  general  avionics  testing 
environment,  the  Navy,  Air  Force,  and  DoD  approaches  to  avionic  testing,  and  some  of 
the  available  government  and  contractor  facilities.  The  final  subsection  deals  with  specific 
Project  Reliance  recommendations.  Section  IV  summarizes  the  principal  cqubilities  of 
ACETEF  and  ECIT,  differentiating  between  those  that  are  existing  or  fund^  through  this 
fiscal  year,  those  that  are  programmed  and  approved  but  not  yet  built,  and  those  that  are  not 
yet  iq;>proved. 

Section  V  covers  Navy  and  Air  Force  testing  requirements,  including  forecasting 
concerns  and  current  Service  estimates,  and  IDA  sensitivity  analyses  of  those  estimates. 
Section  VI  summarizes  the  woridoad  in  terms  of  effectiveness  and  related  marginal  costs 
for  each  oi  the  ACETEF  aq)ability  alternatives.  Costs  include  new  (unsunk)  investmoit  in 
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facilities  and  equipment,  the  additional  operating  costs  of  new  capabilities,  and  deployment 
costs  associated  with  the  Services'  facilities  under  appropriate  alternatives.  We  also  discuss 
the  likely  impact  on  the  Navy  and  Air  Force  if  the  Air  Force  becomes  the  major  ACETEF 
customer.  Section  VII  describes  other  considerations  that  affect  EC  testing,  including 
Service-unique  requirements,  future  workload,  the  Project  Reliance  time  horizon,  and  the 
possible  need  for  cost  effective  albeit  duplicative  facilities.  Section  Vm  presents  findings 
resulting  from  our  analysis  of  the  various  ACETEF  and  ECIT  alternatives. 
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11.  FRAMEWORK  FOR  ANALYSIS 


Testing  of  electronk  combat  capability  and  associated  avionics  (the  two  are 
becoming  more  and  more  integrated)  is  a  highly  technical  and  increasingly  complex 
process.  Comparing  capabilities  across  facilities  can  be  very  subjective  and  controversial 
even  among  recognized  experts.  Estimating  future  EC  requirements  is  imprecise  at  best 
because  of  acquisition  program  uncertainty,  unknown  user  demand  (perhaps  due  to  a  lack 
of  knowledge  as  to  specific  EC  test  capabilities),  program  schedule  uncertainty,  and 
questions  as  to  facility  capacity.  Because  of  variations  in  these  parameters,  test 
requirements  as  well  as  cost  estimates  become  increasingly  imprecise  as  the  projection 
period  expands  into  the  next  century. 

In  establishing  the  analytical  framewoik  for  our  analysis,  we  attempted  to  build  a 
simple  but  comprehensive  structure  that  would  capture  needed  information  and  that  would 
reasonably  acconunodate  the  uncertainties  described  above.  The  framework  was  designed 
to  measure  demands  for  electronic  combat  test  resources  and  the  alternative  sources  of 
supply  to  meet  that  demand.  We  measured  and  compared  demand  and  supply  in  order  to 
determine  effectiveness  and  assign  costs  to  the  various  alternative  ccxnbinations. 

Demand  was  established  by  identifying  aircraft  program  EC  test  needs  requiring 
installed  systems  test  facilities  (Section  A).  The  supply  of  test  resources  was  determined 
by  reviewing  each  of  the  ACETEF  and  ECTT  alternatives  and  measuring  how  each  might 
satisfy  demand.  Our  analysis  focused  on  the  additioiud  costs  incurred  in  acquiring  the  test 
resources  and  was  indifferent  to  the  source  of  fimding  (institutional  or  customer)  (Section 
B).  We  identified  the  resource  requirements  involved  in  EC  testing  (facility  investment, 
facility  opCTatitms  and  test  program  deployment)  and  estimated  their  costs  (Section  C).  We 
used  other  relevant  work  to  supplonent  both  Service-provided  data  and  our  analysis 
(Section  D). 

A.  EC  TEST  REQUIREMENTS 

During  their  acquisition  life  cycle,  individual  aircraft  programs  require  varying 
degrees  of  develofnnent  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  Such  testing  is  performed  eithn  in  flight  or  in  ground  test,  at  either  government 
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or  contractor  facilities.  The  amount  of  ground  testing  for  electronic  combat  depends  upon 
the  expected  use  of  the  aircraft  platform,  as  well  as  the  complexity  (magnitude  and 
intricacy)  and  configuration  (integrated  versus  federated)  of  the  hardware  and  software. 
The  use  of  a  specific  ground  facility  depends  on  the  test  scenario  and  the  size  of  the  aircraft. 

This  study  specifically  dealt  with  installed  systems  test  facilities  (ISTFs)  and  one  of 
their  major  elements,  the  anechoic  chamber.  ISTFs  consist  of  test  equipment,  threat 
simulators  and  stimulators,  and  automated  data  processing  systems  to  perform  EW  ground 
testing  on  an  aircraft  located  in  an  anechoic  chamber.  Anechoic  chambers  provide  a 
shielded  free-space  environment  without  interference  from  external  electronic  emissions 
(including  second'party  monitoring).  For  purposes  of  this  analysis,  requirements  for 
electronic  combat/avionics  ground  testing  were  limited  to  those  for  installed  systems  test 
facilities  that  have  various  laboratories  coupled  with  an  anechoic  chamber.  Sections  ni  and 
IV  describe  the  above  subjects  in  greater  technical  detail,  while  Section  V  describes  the 
requirements  process  and  provides  specific  needs  by  Service  and  aircraft. 

B .  ALTERNATIVE  SOURCES  FOR  ISTF  TESTING 

For  this  study,  we  assumed  the  Air  Force  and  Navy  demand  for  ISTF  testing  can 
be  satisfied  by  capabilities  represented  by  various  configurations  of  facilities,  equipment, 
and  people  through  either  ACETEF  or  ECIT.  Their  potential  ISTF  capabilities  fall  into 
three  classifications:  (1)  the  current  program  (status  quo)  represents  capabilities  that  already 
exist  or  have  been  budgeted,  (2)  programmed  and  approved  capabilities  consist  of  planned 
future  capabilities  sanctioned  in  the  DoD  Future  Years  Defense  Program  (FYDP),  and  (3) 
other  potential  capabilities  that  have  not  been  approved  but  that  are  required  and  reasonable 
additions  within  a  specific  alternative  in  order  to  satisfy  EC  test  requirements 

Funding  can  be  institutional  in  the  form  of  Central  Test  and  Evaluation  Investment 
Program  (CTEIP)  or  Improvement  and  Modernization  (I&M)  investments.  Funding  can 
also  be  direct,  meaning  specific  enhancements  required  and  fimded  by  individual  programs. 
The  category  of  potential  capabilities  contains  additional  capabilities  that  may  be  desirable 
but  have  not  been  approved.  The  ACETEF  and  ECIT  current,  approved,  and  potential 
capabilities  are  summarized  in  Table  1.  We  also  identified  the  probable  sources  of  funding 
to  ensure  that  all  relevant  costs  were  included. 

Using  the  capability  structure  and  information  obtained  from  the  Navy  and  Air 
Force,  we  developed  four  alternatives  for  cost  and  effectiveness  analysis:  (1)  maintaining 
the  current  ACETEF  with  approved  improvements  and  increasing  shift  capacity  to  2.S,  (2) 
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building  a  large  anechoic  chamber  with  both  the  small  and  large  chambers  operating  at  2 
shifts  each,  (3)  upgrading  the  large  chamber  at  Edwards  AFB  to  the  ECIT  configuration 
and  operating  it  at  2.5  shifts,  and  (4)  doing  both  Alternatives  2  and  3.  Each  alternative, 
related  analysis,  and  results,  including  several  excursions  to  incorporate  sensitivity 
analyses  within  each  alternative,  are  presented  in  Section  VI. 

Table  1.  Currant  and  Planned  Capabilities 

ACETEF  ECU 

Current  Small  anechoic  chamber  (SAC),  the  Benefield  Anechoic  Facility  (BAF)  with 

Aircraft  Anechoic  Test  Facility  (AATF)  limited  ISTF  capability 
^)proved  Various  upgrades  through  funding  of  None,  by  direction  of  Congress 

CTTEIP,  I&M 

Potential  Large  anechoic  chamber  (LAC)  through  Various  ISTF  capabilities  through 

_ institutional  funding _ institutional  and  program  funding _ 


C.  TYPES  OF  COSTS 

Providing  necessary  test  resources  such  as  facilities,  materiel,  labor,  etc.,  is  a 
function  of  both  the  program  office  customer  who  generates  the  demand  and  the  test 
organization  that  oversees  the  entire  test  supply  process.  We  developed  our  cost  structure 
to  identify  the  general  type  of  activity  being  performed  and  the  specific  kinds  of  resources 
(cost  elements)  required.  This  structure  was  particularly  useful  in  collecting  all  relevant 
costs.  Specific  cost  elements  were  summarized  by  (1)  investment  costs  to  obtain  the 
needed  capabilities,  (2)  operating  costs  to  include  additional  expenses  of  running  the 
enhanced  test  facility,  and  (3)  deployment  costs  for  people  and  materiel  that  would  be 
moved  to  a  test  location  other  than  that  normally  used  by  that  Service  (i.e.,  AF  programs  to 
Patuxent  River  NAS,  Navy  programs  to  Edwards  AFB). 

As  shown  in  Table  2,  we  identified  four  major  types  of  activities  that  must  be 
accomplished  on  every  test:  (1)  the  test  organization  providing  basic  capability,  (2)  the  test 
organization  providing  program-specific  facilities  and  equipment  as  funded  by  the  program 
office,  (3)  the  test  organization  and  program  personnel  developing  and  performing  the 
actual  test,  and  (4)  the  test  organization  supporting  the  program  test  (indirect  support).  The 
table  also  shows  the  major  cost  elements  or  resource  type  being  used,  e.g.,  facilities,  labor, 
etc.,  for  each  activity  by  investment,  operations  and  (teployment 

All  costs  are  marginal  (also  referred  to  as  differential  or  incremental  costs)  i.e.,  they 
are  additional  costs  (change  in  total  costs)  that  would  be  incurred  when  comparing  which 
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investment  plan  would  yield  the  best  results.  Costs  already  incurred  for  existing  resources 
(e.g.,  facilities,  equipment)  were  considered  to  be  sunk  costs  and,  accordingly,  were  not 
included  in  the  analysis. 


Table  2.  Primary  Cost  Structure 

Investmoit  Operating  and  Etevekwiinait 

Facilities  Materiel  Other  Labor  Travel  Materiel  Other 

lYovide  genetic  test  X  X  X  X  X  X  X 

cqiabilities 

Provide  progratn-specific  XXX 
investment 

Perform  program-specific  X  X  X  X 

test 

Support  program-specific  X  X  X  X 

test 


D.  OTHER  STUDIES  AND  CONSIDERATIONS 

The  general  subjects  of  electronic  warfare  testing,  installed  test  facilities,  and 
anechoic  chambers  have  been  studied  extensively  in  recent  years  in  terms  of  requirements, 
capabilities,  and  costs.  The  information  compiled  in  three  major  study  efforts  was 
particularly  relevant  and  timely.  First,  the  "Final  Report  on  Electronic  Warfare  Test 
Capabilities"  prepared  by  the  Electronic  Warfare  Reliance  Study  Group  [3]  contained  ISTF 
capabilities  and  facilities,  estimated  EC/avionics  test  requirements  (categorized  by  the  size 
of  required  anechoic  chambers),  and  included  recommendations  regarding  future  growth 
and  lead  Service  responsibilities. 

The  Georgia  Tech  Research  Institute  (GTRI)  study  estimated  Navy  ISTF 
requirements  for  the  ACETEF  facility  by  type  of  aircraft  for  FY  1994  through  FY  1996  [7]. 
Finally,  the  OSD  EW  T&E  Investment  Strategy  Panel  was  formed  to  develop  a  coordinated 
DoD  strategy  to  achieve  an  accq)table  balance  in  EW  test  capabilities.  The  panel  consists  of 
representatives  from  the  offices  of  the  Director,  Test  and  Evaluation,  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications  and  Intelligence,  and  the 
Director  of  Operational  Test  and  Evaluation  and  each  of  the  Services.  To  date,  the  panel 
has  issued  two  reports,  which  outline  the  panel's  findings  and  make  recommendations  to 
improve  the  EC  testing  process  [5  and  6].  The  study  team  also  reviewed  some  SO  other 
documents  [7  through  56]. 
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III.  ELECTRONIC  COMBAT/AVIONICS  SYSTEM  TESTING 


Installed  systems  test  facilities  (ISTFs)  are  complexes  of  anechoic  chambers, 
automated  data  processing  systems,  standard  and  unique  test  equipment,  threat  simulators, 
and  other  devices  necessary  to  simulate  mission  scenarios  and  stimulate  an  EC/avionics 
system  installed  and  integrated  in  an  operationally  functional  configuration.  Over  the  last 
decade,  the  use  of  ISTFs  has  grown  as  their  utility  has  been  recognized  in  trouble-shooting 
systems  under  test  as  well  as  serving  as  an  integral  part  of  both  development  test  and 
evaluation  and  operational  test  and  evaluation.  The  congressional  language  that  generated 
this  study  did  not  challenge  the  utility  of  instaUed  systems  test  facilities.  Nonetheless,  we 
believe  that  a  review  of  modem  EC/avionics  system  trends  and  the  associated  test 
complexities  will  facilitate  understanding  later  portions  of  this  study  that  address  test 
requirements  and  the  cost  and  effectiveness  of  alternative  approaches. 

A.  BACKGROUND 

1.  Trends  in  Electronic  Combat/Avionics  Systems 

There  is  a  wealth  of  anecdotal  evidence  that  the  complexity  of  modem  avionics 
systems,  as  well  as  the  complexity  of  testing  them,  is  increasing  rapidly.  From  a  cost 
perspective,  EC/avionics  subsystems  represent  a  growing  fraction  of  the  procurement  cost 
of  new  aircraft — ^approaching  one-third  of  systems  currently  in  production  (see  Figure  13 
in  Section  VII).  The  next  generation  of  aircraft  (B-2,  F-22,  A-X)  are  expected  to  continue 
this  trend. 

In  addition,  the  complexity  of  avionics  software,  as  measured  by  lines  of  code,  is 
rapidly  increasing,  with  the  newest  systems  requiring  over  one  million  lines  of  code,  and 
the  next-generation  of  systems  expected  to  require  even  larger  software  programs  (see 
(Figure  14).  Overall,  the  contribution  of  avionics  systems  and  software  to  system 
effectiveness  has  grown  rapidly,  and  with  it  the  importance  of  thorough  testing  under 
conditions  that  stress  the  system  in  a  fashion  representative  of  the  likely  operational 
environment 
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A  specific  example  is  the  F-16  flight  test  program,  shown  in  Figure  1  from  its 
prototype  phase  in  the  mid-1970s.  First,  note  that  significant  testing  continued  long  after 
production  began  in  mid- 1978.  There  have  been  more  test  flying  hours  in  virtually  every 
year  since  then  than  there  were  in  any  year  prior  to  the  start  of  production.  Second, 
because  of  the  increasingly  complex  avionics  modifications  and  upgrades  reflected  in  the 
block  designations  along  the  horizontal  axis,  the  number  of  test  flight  hours  have  moved 
consistently  up.  The  leveling  of  hours  in  the  late  1980s  can  be  attributed  to  the  activation  of 
the  current  F-16  Integrated  Facility  for  Avionics  Systems  Test  (IFAST)  capability  at 
Edwards  AFB.  Testing  a  current  F-16  avionics  tape  (a  software  program)  can  involve 
almost  1,000  different  test  information  sheets  (TISs),  each  of  which  represents  a  set  of 
conditions  under  which  the  whole  system,  with  its  software,  is  to  be  tested  one  or  more 
times.  This  compares  to  several  hundred  TISs  as  recently  as  the  mid-1980s. 
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Figure  1.  Total  F-16  Test  Support  Flight  Time 


2.  The  Trend  Toward  Highly  Integrated  Systems 

Perhaps  more  dramatic  than  the  general  increase  in  avionics  system  hardware  and 
software  complexity  is  the  quantum  leap  in  complexity  that  accompanies  the  new  highly 
integrated  EC/avionics/flight  controls/mission  computer  systems.  The  F-16  Block  SO,  the 
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F-ISE,  and  the  MC-130  have  already  started  the  trend  toward  greater  integration.  The  B-2, 
C-17,  F-22,  and  RH-66  will  have  the  first  of  the  fully  integrated  EC/avionics/flight 
controls/mission  computer  systems.  The  implications  for  testing  are  significant  In  the 
previous  generation  of  largely  federated  systems,  specific  functions  were  performed  by 
individual  "black  boxes"  and  software.  This  permitted  individual  functions  and  the 
subsystems  that  performed  them  to  be  tested  in  isolaticm  from  the  rest  of  the  system. 

In  integrated  avionics  systems,  the  various  functions  are  still  associated  with 
identifiable  "sensors,"  but  the  computational  process  is  performed  by  hardware  and 
software  residing  in  multi-purpose  black  boxes  within  the  total  system.  Which  black  box 
performs  a  function  changes  with  the  overall  demands  being  placed  on  the  avicmics  system 
as  a  whole.  Testing  a  "subsystem"  thus  requires  that  the  entire  avionics  package  be 
operating,  with  representative  simulations  stressing  the  various  functions,  and  with  the 
whole  system  subjected  to  some  form  of  simulated  mission  or  operation,  whether  via  flight 
test  or  ground  test 

3 .  The  Need  for  a  Mix  of  Flight  and  Ground  Test 

The  test  process  ultimately  serves  an  array  of  objectives.  Early  in  a  test  program, 
the  emphasis  will  be  placed  on  getting  the  system  to  work.  This  involves  ensuring  that  the 
components  fit  together  and  generally  operate  in  the  intended  fashion.  Much  of  this  phase 
is  trouble-shooting.  Later,  the  emphasis  changes  to  measuring  system  performance  for 
selected  parameters  or  characteristics  in  order  to  determine  compliance  with  contractual 
requirements.  Overlapping  this  compliance-oriented  testing  and  continuing  into  operational 
employment  is  testing  to  determine  military  effectiveness  and  suitability,  to  refine  tactics, 
etc. 

Performing  all  the  testing  required  for  modem  avionics  systems  by  flight  test  is 
simply  not  achievable.  The  F-16  Combined  Test  Fence  estimated  that  up  to  80%  of  the  test 
runs  (basically,  one  TIS  at  a  specified  set  of  test  conditions)  can  be  performed  on  the 
ground.  One  can  imagine  how  Hgure  1  would  look  in  the  absence  of  ground  testing. 

Overall,  the  case  for  ground  testing  as  an  adjunct  to  flight  testing,  tests  on  several 

factors. 

•  Ground  tests  can  have  a  cost  advantage  of  between  7  and  IS  to  1  over  flight 
test  [54].  Greater  knowledge  about  the  performance  of  the  system  can  be 
gained  at  lower  cost,  and  probably  in  less  time,  with  reduced  risk  to  the  test 
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article  and  the  flight  crews.  More  importantly,  certain  tests  that  simply  are  not 
practical  in  flight  can  be  performed. 

•  The  complexity  and  integrated  nature  of  modem  avionics  systems  requires  that 
they  be  thoroughly  checked  in  a  forgiving  environment  before  risking  flight 
test.  Only  a  ground  test  facility  capable  of  operating  the  full  system  under 
rigorous  stresses  representative  of  what  the  system  will  see  in  flight  is  adequate 
to  perform  this  pre-flight  test  activity. 

•  Modem  systems  are  designed  to  operate  in  such  a  wide  array  of  conditions, 
with  so  many  operating  modes,  that  testing  all  via  flight  test  would  be 
prohibitively  expensive. 

•  The  information  gained  from  ground  test  can  be  used  to  focus  flight  test  on 
regimes  that  will  give  a  thorough  evaluation  of  the  system. 

•  Many  aspects  of  the  test  process  require  that  large  numbers  of  signals  at  high 
data  rates  be  monitored  on  a  near  continuous  basis.  Physical  constraints  within 
an  aircraft  in  flight  make  this  almost  impossible.  The  difficulty  is 
compounded  by  the  inability  to  precisely  control,  or  even  measure, 
environmental  conditions  in  flight.  (While  eventually  the  aircraft  will  need  to 
operate  in  whatever  environment  happens  to  exist,  large  portions  of  the  flight 
test  program  require  either  a  controlled  or  a  known  environment  so  as  to 
evaluate  the  response  of  the  system  to  known  stimuli.). 

•  It  is  often  difficult  to  present  sufficient  threat  or  target  density  on  an  open  air 
range. 

•  Finally,  security  concerns  argue  against  operating  certain  threat  replicas,  and 
"blue"  systems  responses,  on  an  open  air  range. 

4.  The  Need  for  Installed  Systems  Test  Facilities  (ISTF) 

A  wide  array  of  ground  test  facilities  already  exist.  The  IFAST  at  Edwards  AFB, 
already  discussed  briefly,  is  basically  a  "spread  bench"  for  avionics  to  be  laid  out  in  a 
fashion  that  facilitates  stimulation  of  parts  of  (or  all  of)  the  system,  and  measurement  of  the 
system  response.  However,  the  system  is  not  installed  in  the  airframe,  and  many  radiating 
components  are  not  able  to  be  operated  in  a  fully  functional  configuration.  These 
deficiencies  drive  the  need  for  ISTF  class  facilities. 

An  ISTF,  by  definition,  tests  the  avionics  system  in  its  "installed"  configuration. 
The  presence  of  an  anechoic  chamber  (or.  for  limited  purposes,  a  shielded  hangar)  permits 
radiating  elements  of  the  system  to  radiate  without  fear  of  hostile  interception.  The 
chamber  effectively  isolates  the  system  under  test  from  outside  radiating  sources,  which 
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facilitates  relating  system  responses  to  known  stimuli.  Finally,  depending  upon  the  test 
concept,  the  anechoic  chamber  may  permit  use  of  radiating  threat  simulators. 


ISTFs  fall  into  three  categories,  based  upon  the  sophistication  of  the  test  scenarios 
and  the  fidelity  of  the  threat  simulators.  Based  upon  definitions  taken  from  the  EW 
Reliance  Study  Group  [3],  the  categories  are  defined  as  follows: 

•  Category  I  facilities  provide  mission-level  testing  using  test  article  and 
validated  red/blue/gray  environments  (threat/simulation)  with  free  play  (both 
threat  and  test  article)  operators  in  realistic,  large-scale  engagements. 

•  Category  II  facilities  provide  platform  level  testing  with  free  play  (test  article 
only)  operators  in  the  loop  and  limited,  validated  red/blue/gray  stimulation  in 
realistic  scenarios. 

•  Category  III  facilities  provide  system  or  subsystem-level  testing  using 
minimal,  calibrated  stimulation  of  test  articles  under  specific  conditions. 

•  The  Reliance  Study  also  identified  a  fourth  category,  which  provides  special- 
purpose  testing.  This  category  includes  radar  cross-section  measurement 
facilities,  antenna  pattern  measurement  facilities,  and  other  facilities  that  test 
less  than  complete  avionics  systems  or  limited  characteristics  of  the  test  article. 

B .  SERVICE  APPROACHES  TO  TESTING  AVIONICS 

The  two  principal  facilities  under  review  in  this  study  are  two  Category  II  facilities: 
the  programmed  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF)  at 
Patuxent  River  NAS,  and  the  proposed  Electronic  Combat  Integrated  Test  (ECIT)  facility  at 
Edwards  AFB.  To  an  extent,  the  actual  use  of  ACETEF,  and  the  proposed  use  of  ECIT, 
reflect  slightly  different  approaches  to  avionics  systems  test  Both  the  Air  Force  and  the 
Navy  tend  to  view  ground  testing  as  serving  two  principal  roles:  (1)  it  provides  a  low-risk 
environment  in  which  to  get  the  system  under  test  "up  and  running,"  and  (2)  it  permits  the 
system  to  be  run  through  its  paces  throughout  its  entire  performance  envelope,  serving  as  a 
screening  device  for  focusing  subsequent  flight  testing  on  those  portions  of  the 
performance  envelope  perceived  as  potential  problem  areas. 

Having  identified  these  shared  views,  however,  we  also  believe  that  the  Navy  has 
tended  to  view  a  visit  to  the  ACETEF  as  a  relatively  infrequent  event  for  the  typical 
program.  Put  another  way,  a  system  comes  to  ACETEF  prior  to  significant  flight  test  for  a 
relatively  lengthy  stay  (weeks  to  months).  During  this  time,  the  system  is  thoroughly  run 
through  its  paces,  and  every  problem  that  can  be  identified  and  fixed  on  site  is  fixed.  The 
system  then  goes  into  flight  test,  often  at  a  location  remote  from  Patuxent  River.  The 
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system  may  return  to  AGSTEF  at  some  future  point,  but  these  visits  are  expected  to  be  Inief 
aiKl  widely  separated  in  time. 

The  Air  Force,  on  the  other  hand,  seems  to  intend  to  use  the  ECIT  (assuming  it  is 
pursued)  in  a  fashion  somewhat  analogous  to  the  IFAST.  Systems  arriving  for  flight  test 
would  go  to  ECIT  for  as  comprehensive  testing  as  possible,  with  problems  fixed  along  the 
way.  When  sufficient  confidence  has  been  established  in  the  functionality  of  the  system, 
flight  test  would  start  at  the  same  installation  as  the  ECIT  ground  test,  and  supported  by 
many  of  the  same  test  engineers  (both  contractor  and  government).  The  plan  would  be  for 
frequent,  ^(xt  returns  to  the  ECIT  to  "clear"  increasingly  complex  portions  of  the  avicmics 
system  for  flight  test  This  approach,  theoretically,  places  a  high  premium  on  collocation 
of  the  flight  test  program  with  the  ISTF  (in  this  case,  the  ECIT). 

There  are  numerous  factors  that  drive  these  approaches.  First  the  most  extensive 
threats  and  threat  simulators  are  located  on  ranges  in  the  southwest  United  States,  on  or 
near  Edwards  AFB.  The  threats  of  most  interest  to  the  Navy  are  located  at  China  Lake, 
CA,  far  away  from  Patuxent  River,  MD. 

It  is  not  clear  that  either  the  Navy  or  Air  Fmce  ^proach  is  "right"  The  approaches 
appear  to  reflea  historically  different  methods  that  have  been  successfully  used  by  the  two 
Services. 

C.  AVAILABLE  AVIONICS  GROUND  TEST  FACILITIES 

For  purposes  of  this  study,  ground  test  facilities  need  an  anechoic  chamber  at  least 
large  enough  to  test  a  tactical-sized  aircraft  For  our  purposes,  we  followed  the  Navy 
terminology  and  called  this  a  "small  anechoic  chambor."  (The  DoD  EW  Reliance  Study 
Group  calls  this  size  of  chamber  "intermediate"  [3].)  In  addition,  the  DoD  EW  Test  and 
Evaluation  Investment  Strategy  report  found  Aat  free-q)ace  radiation  usability  in  a  multi- 
spectral  [electro-t^tical  (EO),  infrared  (IR),  and  radio  frequency  (RF)]  facility  best 
addresses  the  requirements  for  inlegiated  system  test  [S].  No  existing  facility  fully  satisfies 
the  requirement  at  this  time.  However,  the  ACETEF  comes  close,  and  programmed 
enhancements  (Le.,  those  that  are  pan  of  an  ap|»oved  FYDP)  will  bring  it  closer. 

Hie  PkefUght  iMegratioa  Munitions  and  Electronic  Systems  (PRIMES)  facility  at 
EgUn  AFB  represents  a  significant  capdiility  for  integration  of  armament  systems  with  host 
aircraft  and  for  test  of  EC  syMems.  However,  it  does  not  currently  have  a  significant 
capability  for  imegraied  avionics  systems  test  In  addition,  the  PRIMES  facility  was  not 
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iiKluded  in  the  congressional  language  directing  this  study.  We  have  therefore  not 
considered  the  PRIMES  as  a  i»incipal  candidate  installed  EC/avionics  systems  test 

The  current  ECIT  at  Edwards  AFB  represents  a  limited  capability  from  an  ISTF 
perspective.  With  enhancements  as  described  by  the  Air  Force  during  this  study,  the 
facility  could  achieve  a  significant  capability — comparable  to  the  ACE'i  tF  as  currently 
programmed.  There  are,  however,  technical  risks  associated  with  the  q^roaches  imposed 
by  both  the  Navy  and  Air  Force  for  enhancing  their  respective  facilities,  which  are 
discussed  later  in  this  report 

We  are  aware  of  only  Category  m  ISTF-class  facilities  at  contractor  plants,  such  as 
General  Dynamics-Fort  Worth  and  Grumman.  Most  contractors  who  are  heavily  involved 
in  EC/avionics  work  have  systems  integration  laboratories  (SILs),  which  represent  a 
significant  "bench  integration  and  test"  capability. 

D.  PROJECT  RELIANCE  AND  THE  DOD  EC  STRATEGY 

Project  Reliance  is  an  outgrowth  of  Defense  Management  Review  Directive  922, 
issued  in  late  1989,  directing  changes  in  management  of  Defense  R&D  activities.  It  is  a 
joint’Service  effort,  under  the  direction  of  the  Joint  Logistics  Commanders  Joint 
Coordinating  Group  for  T&E.  The  objective  of  Project  Reliance  is  to  eliminate 
unnecessary  duplication  while  better  focusing  limited  T&E  investment  funding  on 
improvement  and  modernization  projects  diat  provide  the  best  overall  gain  in  test  cq>ability 
for  the  DoD. 

Reliance  reviewed  existing  ISTF  capabilities  and  projected  DoD  requirements  fw 
such  facilities.  The  conclusions  they  reached  that  are  rdevant  to  diis  study  were: 

•  1.4  Category  1, 4.5  CategtMry  n,  and  3.8  Category  m  facilities  are  required. 
(AECTEF  is  currently  a  Category  n  facility,  PRIMES  and  the  possible  ECIT 
are  Category  m  facilities). 

•  3.2  large  aircraft  anechoic  chambers  are  required.  (There  now  is  tme,  with 
limited  ISTF  capability.) 

The  study  designated  the  Navy  as  lead  for  ISTFs,  and  recommended  that  the 
Services  under  Navy  leadership  develop  an  ISTF  Master  Plan.  The  Reliance  study  group 
recommended  that  the  following  be  omsideied: 

•  ACETEF  be  developed  to  a  Categwy  I  level  of  capability,  widi  an  (mcisting) 
fightn-sized  chamber.  Effort  on  a  large  chambm*  diould  be  limited  to  planning 
activities,  pending  clarification  of  ISTF  workload.  The  current  CTEIP 
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program  is  w(»ldng  toward  this  mid,  with  lessons  learned  to  be  shared  by 
subsequent  developers. 

•  The  ECIT  (the  Edwards  facility,  referred  to  as  ECIT  in  the  Reliance  study) 
should  be  developed  into  a  Category  n  facility  principally  supporting  new- 
generation  systems.  Alternative  3  of  this  study  follows  this  Reliance 
reconunendadmi. 

•  PRIMES  should  be  developed  into  a  Category  n  facility  primarily  to  support 
armament  integration  and  test  of  aormit-generation  systems. 

Data  submitted  by  the  Services  in  support  of  this  IDA  study  reduced  the  workload 
from  the  projected  levels  used  in  the  Project  Reliance  study.  For  example,  the  AF  reduced 
its  projection  over  33%.  The  Service  data  provided  requirements  for  more  ISTF  capacity 
than  currently  exists,  particulariy  the  need  for  a  large  chamber.  Section  V  provides  details 
of  the  ISTF  requirements. 
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IV.  ACETEF  AND  ECIT  CAPABILITIES 


This  section  of  the  report  provides  a  summary  of  current  capabilities  at  the  two 
facilities  that  are  the  primary  focus  of  this  study — ^the  Navy's  ACETEF,  including 
programmed  enhanc^ents;!  and  the  Air  Force's  ECIT  as  it  currently  exists.^  This  section 
also  summarizes  possible  additional  enhancements  at  ACETEF  and  at  ECIT  that  will 
provide  additional  ISTF  capacity  and  capability.  Later  sections  address  the  costs  and 
benefits  of  these  possible  additional  enhancements. 

A.  CURRENT  CAPABBLITIES 

1 .  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF)^ 

The  ACETEF  is  located  at  the  Naval  Air  Warfare  Center,  Patuxent  River  NAS, 
Maryland  [formerly  known  as  the  Naval  Air  Test  Center  (NATC)].  This  organization  is 
generally  recognized  within  the  DoD  electronic  combat/avionics  test  community  as  having 
the  broadest  experience  in  developing  and  using  all  categories  (i.e.,  I  through  IV,  as 
discussed  earlier)  of  ISTF/anechoic  chambers,  especially  in  the  areas  of  installed  systems 
testing  and  electromagnetic  envircmmmital  effects  (E^).^ 


^  We  treated  the  progmanedcalnncemeais  as  "given”  ID  this  study,  because  ibey  are  pvt  of  an  qjproved 

FYDP  piogrvn  vid  financial  {dan.  Nothing  in  the  congiessioiial  committee  language  initiating  this 
study  indicaies  that  Congress  queslioos  the  programmed  ACb  lliF  enhancemcwts. 

2  The  existu^groimd  test  facilities  tt  the  Air  Force  Flight  Test  Center  (AFFTC)  consist  of  the  Test  and 
Evaluatkm  Mission  Simulator  (lEMS).  the  Integration  Facility  for  Avionics  Systems  (IFAST),  and 
the  Benefield  Anechoic  Facility  (BAF).  EOT  is  the  term  a^died  to  these  bcilities  plus  addidonal 
simulation,  stimulation,  and  command  and  control  capaldlities.  For  ease  ctf  we  use  the 

term  EOT  to  describe  the  facilities  now  existing  VAHKIC,  as  weB  as  the  enhanced  facilities  discussed 
later  in  the  section. 

^  Most  of  ttedeao^Nion^  Navy  ground  lest  faculties  was  taken  fiom  an  undated  NATC  pamphlet.  "Air 
Combtt  Environment  Test  and  Evalnaiion  Facility  (ACETEF),"  which  was  distributed  during  a 
briefasg  to  the  IDA  study  team  (see  Reference  [2]).  This  panqddet  discusses  Navy  ground  test  ftom  the 
early  1970s  through  the  current  ACETEF'  oonc^ 

*  Baaed  on  narrative  induded  in  Reference  [3],  p  33. 
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a.  General  Description 

The  origins  of  the  Navy's  ground  test  facility  program  can  be  traced  to  the  1970s, 
when  the  then  NATC  developed  six  largely  autonomous  simulation/stimulation  laboratories 
for  ground  test  of  aircraft  subsystems  and  systems.  The  laboratories  were  intended  to 
support  flight  test  by  ground  testing  the  systems  throughout  as  much  of  their  performance 
envelope  as  possible.  The  laboratories  were: 

•  Tactical  Avionics  Software  Test  and  Evaluation  Facility  (TASTEF).  Through 
simulation  and  hardware-in-the-loop,  TASTEF  allows  engineers  to  evaluate 
core  avionics  hardware  and  software  in  a  controlled  ground  environment 

•  Electronic  Warfare  Integrated  Systems  Test  Laboratory  (EWISTL).  This 
permits  radio  frequency  (RF)  stimulation  of  EC  equipment  through  interfaces 
at  the  antennae.  EWISTL  can  simulate  1,024  radar  emitters  with  pulse 
densities  of  4  million  pulses  per  second. 

•  Aircrew  Systems  Evaluation  Facility  (ASEF).  This  primarily  focuses  on 
controls  and  displays  and  is  used  to  evaluate  the  man-machine  interface. 

•  Antenna  Test  Laboratory  Automated  System  (ATLAS).  ATLAS  provides  the 
capability  to  develop  antenna  pattern  data  from  aircraft  flight  data.  The  data  is 
used  by  RF  generating  facilities  to  model  antenna  effects  when  the  system  is 
stimulated  by  signal  injection  behind  the  antenna. 

•  Navy  Identification  Friend  or  Foe  Test  and  Evaluation  (NIFFTE).  This  is  a 
hardware-in-the-loop  facility  for  testing  identification  friend  or  foe  (IFF) 
subsystems. 

•  Electromagnetic  Environment  Generating  System  (EMEGS).  This  facility 
simulates  32  high-powered  emitters  to  simulate  the  electromagnetic 
environment  on  the  deck  of  an  aircraft  carrier. 

By  the  early  1980s,  examination  of  critical  test  and  evaluation  issues  led  the  Navy 
to  the  conclusion  that  significant  changes  were  needed  in  its  approach  to  ground  test  The 
issues  were  test  realism,  threat  realism  (both  density  and  Hdelity),  security,  increasingly 
integrated  test  articles,  interoperability  (among  systems),  and  cost/complexity  of  the  test 
articles  as  well  as  the  test  facilities.  The  result  of  this  examination  was  development  of  the 
ACETEF  program,  shown  conceptually  in  Hgure  2,  which  will  be  achieved  over  a  period 
of  time  as  funding  allows. 


18 


Figura  2.  ACETEF  Program 


ACETEF  achieved  an  initial  capability  in  1983,  when  the  anechoic  chamber  was 
placed  into  operation.  The  chamber  is  60  feet  wide  by  100  feet  long  by  40  feet  high.^  It 
has  a  30-ton  crane  in  the  ceiling,  has  a  40  foot  (width)  by  20  foot  (height)  door,  and 
provides  120  decibels  of  isolation.  Any  aircraft  system  except  the  propulsion  system  can  be 
operated  within  the  chamber.  The  current  ACETEF  also  better  integrates  and  coordinates 
the  laboratories  that  already  existed,  and  provides  the  following  additional  capabilities: 

•  A  closed-loop  facility  provides  an  RF  threat  simulator/stimulator  c{q)ability;  that 
is,  the  threat  simulator  responds  to  the  countermeasures  taken  by  the  system 
under  test 

^  In  the  lenns  used  in  this  study,  the  chamber  is  a  "small"  one,  suitable  for  testing  tacdcal-sized  aircraft 
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•  The  EMEGS  has  been  upgruled  U)  the  Electromagnetic  Environmental  Effects 
Test  Laboratory  (E^TL)  to  provide  higher  power  levels  and  mote  emitters. 

•  EWISTL  is  being  upgraded  to  increase  the  number  and  types  of  emitters, 
expand  the  frequency  range,  and  extend  its  interface  Ciy)abilities. 

•  The  TASTEF  has  been  upgraded  to  provide  a  Manned  Flight  Simulator  that 
puts  the  aircrew  (as  opposed  to  test  engineer)  in  the  loop. 

•  A  variety  of  other  upgrades  have  occurred  to  the  Mission  Planning  Center  and 
other  elemoits  of  the  facility. 

Overall,  the  ACETEF  capability  represents  the  most  comprehensive  ISTF  currently 
in  existence.  However,  the  size  of  the  chamber,  and  its  door,  limit  the  facility  to 
tactical-sized  aircraft  Further,  the  size  of  the  chamber  would  appear  to  limit  stimulation 
methodology  to  signal  injection  and/or  the  use  of  "antenna  hats."  Free-space,  far-field 
radiation  is  not  possible.  The  relevance  of  these  limitations  is  discussed  in  Appendix  G. 

b.  ACETEF  Current  Usage 

ACETEF  usage  has  generally  increased  since  the  activation  of  the  anechoic  chamber 
in  1983.  The  anechoic  chamber  is  currently  the  "limiting  resource" — that  is,  from  a  facility 
point  of  view,  the  total  throughput  of  test  work  at  ACETEF  is  primarily  limited  by  the 
availability  of  chamber  time.  The  chamber  has  been  used  more  than  one  shift  per  day  since 
the  mid- 1980$,  and  recently  has  averaged  about  1.S  shifts  per  day  on  the  basis  of  a  S-day 
work  week  (see  Figure  3).  When  the  chamber  is  being  used,  generally  at  least  one 
supporting  laboratory  will  also  be  needed.  The  actual  number  of  supporting  labs  used  will 
depend  upon  the  complexity  and  degree  of  integration  of  the  system  under  test,  as  well  as 
the  complexity  of  the  test  objectives  associated  with  the  specific  test  run.  At  this  time,  the 
ACETEF  can  generally  support  more  than  orre  customer  at  a  time.  (In  fact,  on  one  day  of 
the  study  team's  visit,  the  manrred  flight  simulator,  the  aircrew  systems  evaluation  facility, 
and  the  anechoic  chamber  were  all  in  use  for  three  different  test  programs.) 

Generally,  ACETEFs  workload  has  consisted  of  test  programs  for  new  and/or 
upgraded  subsystems  installed  in  aircraft  that  are  already  operatitMuiL  No  new  development 
aircraft  have  undergone  testing  at  ACETEF.  The  Navy  also  indicated  that,  with 
maintenance  considerations,  facility  use  could  be  expanded  to  a  maximum  of  2.5  shifts  per 
day. 
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Figure  3.  ACETEF  Workload 

2.  The  Electronic  Combat  Integrated  Test  Facility  (ECIT) 

The  ECrr  is  located  at  the  Air  Force  Flight  Test  Center  (AFFTC),  Edwards  Air 
Force  Base,  California.  AFFTC  is  the  Air  Force’s  principal  facility  for  aircraft  fight  test 
The  ECIT  was  proposed  in  the  FY  1992  President's  Budget  request  for  initiation  of  a  major 
investment  program  to  bring  the  existing  facilities  op  to  the  level  of  an  installed  systems  test 
facility.  Congress  denied  the  request,  and  required  that  the  issue  of  ECIT  cost 
effectiveness,  compared  to  that  achieved  by  additional  investment  at  ACETEF,  be  reviewed 
and  reported  to  the  s^prc^riations  committees. 

a.  General  Description^ 

The  use  of  ground  test  to  support  flight  test  at  AFFTC  can  be  traced  to  the  19S0s, 
when  analog  simulations  supported  the  X-2  and  Dynosoar  programs.  The  current  facilities 
were  largely  activated  during  the  1980s,  and  consist  of  three  principal  elements: 

•  The  Test  and  Evaluation  Mission  Simulator  (TEMS).  The  TEMS  provides 
real-time  piloted  simulations  of  flight  dynamics  and  aircraft  systems.  A  test 
station  typically  includes  a  cockpit,  with  "out  the  window"  scene  generation. 


^  Much  of  tbenuaerial  in  this  sectkm  was  derived  fipon  Reference  [31]. 
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driven  by  a  digital  simulation  of  aerodynamics,  propulsion,  flight  controls  and 
overall  flying  qualities.  Various  mission  simulators  have  been  in  use  since  the 
1950s.  Current  simulations  include  the  F-4E,  F-5F,  Space  Shuttle,  F-15A-E, 
F-16A/C,  A-7D/YA-7F,  B-IB,  and  several  others.  The  simulators  do  not 
provide  base  motion,  and  the  scene  generation  is  rudimentary  compared  to  that 
found  in  contractor  man-in-the-loop  simulators.  Air  Force  laboratories,  or  the 
Navy’s  ACETEF. 

•  The  Integrated  Facility  for  Avionics  Systems  Test  (IFAST).  The  IF  AST 
provides  avionics  "spread  bench"  test  capability  for  functional  verification, 
anomaly  characterization,  integration  testing,  software  evaluation,  and 
aircrew/flight  test  engineer  familiarization.  The  IFAST  was  first  available  for 
use  in  November  1982,  and  reached  its  current  conflguration  in  December 
1988.  The  facility  contains  seven  program  test  areas  of  about  7000  square 
feet,  of  which  6  have  2,800-square-foot  shielded  bays  with  an  outside  field  of 
view,  permitting  operation  of  an  attack  radar,  for  example.  Basically, 
however,  the  IFAST  provides  space,  power,  and  other  utility  services  only; 
any  capability  for  simulation  or  stimulation  to  the  test  article  is  determined  by 
what  the  test  program  decides  to  put  into  the  IFAST.  For  example,  the  F- 16 
occupies  two  bays,  and  has  equipped  both  an  Integrated  Avionics  Test  Station 
(INATS)  and  a  Radar  Test  System  (RTS).  Overall,  the  F-16  represents  the 
most  significant  capability  now  at  IFAST,  and  its  use  has  helped  to 
significantly  reduce  flying  hours,  as  discussed  in  Section  III. 

However,  there  are  inherent  limitations  in  the  IFAST.  The  test  article  is  not 
"installed,"  so  that  the  geometric  configuration  and  cable  runs  are  not 
necessarily  representative  of  the  installed  configuration,  and  effects  of  airframe 
and/or  propulsion  components  on  the  avionics  are  not  represented.  Also,  there 
are  significant  limitations  on  the  ability  to  use  radiating  elements  of  the  system 
under  tesL 

•  The  Benefield  Anechoic  Facility  (B  AFO.  The  B  AF  is  a  large  anechoic  facility, 
located  near  the  IFAST.  It  provides  a  controlled  electromagnetic  environment 
in  which  a  test  article  or  system  can  be  stimulated  and  its  responses  recorded 
and  analyzed.  The  chamber  size  is  264  feet  flength)  by  250  feet  (width)  by  70 
feet  (height),  with  a  door  200  feet  (width)  by  68  feet  (height).  The  chamber  is 
large  enough  to  test  large  aircraft  (B-52,  B-IB,  B-2,  C-17,  C-130,  E-6,  etc.) 
or  "formations"  of  up  to  several  tactical  aircraft;  and  is  theoretically  large 
enough  to  permit  actual  radiation  by  threat  simulators  through  (far  field)  free 
space  to  stimulate  the  system  under  test."^  The  chamber  has  an  80-foot- 
diameter  turntable,  capable  of  rotating  a  250,(XX)-pound  load.  This  permits 
changing  the  angle  of  arrival  of  signals  transmitted  from  three  radiating  threat 
sites  located  at  120-degree  intervals.  One  site  is  equipped  with  a  threat 


^  The  merits  of  this  approach  (free  space  radiaticm,  as  opposed  to  signal  injection)  are  somewhat  disputed 
within  the  technical  community,  as  is  the  suitability  of  the  BAF  design  to  support  such  a  test 
stimulation  methodology.  This  issue  is  discussed  in  greater  depth  in  Appendix  G. 


22 


generator  capable  of  modeling  256  simultaneous  background  threats.  The  other 
sites  have  very  limited  background  threat-generation  capability.  Finally,  an 
80,000-pound  hoist  is  being  installed  in  the  ceiling  of  the  B  AF.  The  B  AF  was 
originally  built  in  June  1989  as  part  of  the  B-IB  program,  and  turned  over  to 
AFFTC  as  a  generic  test  facility  in  September  1990.  When  the  chamber  was 
initially  activated  to  support  B-IB  defensive  avionics  testing,  the  threat- 
generation  capability  was  limited  to  four  (of  eleven)  "Top- 11"  threats.  The 
Areat-generation  capability  has  subsequently  matured  so  that  a  higher  overall 
threat  density  and  all  1 1  of  the  "Top-1 1”  threats  can  be  generated.  However, 
the  BAF  can  generate  neither  the  density  nor  the  number  of  specific,  high- 
priority  threats  that  the  ACETEF  can  provide,  nor  is  there  any  "closed  loop" 
threat  capability.  Finally,  the  BAF  lacks  the  simulation,  simulation,  and 
command  and  control  capabilities  necessary  to  be  a  truly  useful  facility. 

Overall,  the  ECIT  capability  is  not  comparable  to  that  which  exists  today  at 
ACETEF.  Among  other  issues,  the  three  facilities  at  AFFTC  are  not  generally  integrated — 
that  is,  an  aircrew  member  in  the  TEMS  cannot  "fly"  (or  fly  against)  the  avionics  package 
in  the  IFAST,  nor  can  the  simulators  and  stimulators  in  the  IFAST  provide  signals  to  an 
aircraft  in  the  BAR* 

b.  ECIT  Current  Usage 

While  the  IFAST  is  used  productively  to  support  ongoing  tests,  the  BAF  is  of 
limited  utility.  FY  1991  data  extracted  from  the  Range  Utilization  Measurement  System 
(RUMS)  indicates  that  each  IFAST  test  area  currently  occupied  is  in  use  about  one  shift  per 
day,  5  days  per  week.  If  workload  required  it,  and  funding  were  available,  IFAST 
utilization  would  be  increased  to  about  2  shifts  per  day  per  test  area.  The  BAF  is  used 
about  50%  of  a  shift,  although  much  of  the  BAF  use  to  date  has  been  for  chamber 
characterization  and  aircraft  "fit"  checks.  The  capacity  of  the  BAF  is  currently  limited  to 
one  shift  per  day,  based  on  manpower  and  workload,  although  a  limited  surge  capability  to 
2  shifts  a  day  is  possible.  Inherently,  the  chamber  capacity  is  2  to  2.5  shifts  per  day,  but 
additional  personnel  would  be  required.  The  TEMS  has  been  operating  at  about  20%  of 
capacity  recently — but  the  nature  of  this  facility  is  such  that  its  heaviest  use  should  be 
expected  when  wholly  new  aircraft  or  major  model  changes  occur,  events  which  have  not 
occurred  recently. 


*  The  TEMS  and  IFAST  were  integrated  in  1991  for  demonsuation  purposes.  The  F-16  Block  40 
INATS  and  an  F-15  TEMS  crew  station  were  connected  in  "real-time."  However,  this  sort  of 
integratiCHi  is  not  the  norm. 
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Fundamentally,  the  only  portion  of  the  ECIT  that  represents  a  significant,  current 
capability  is  the  IFAST,  and  it  is  heavily  used.  The  remainder  of  the  ECIT  is  of  limited  use 
unless  additional  investment  occurs  to  better  integrate  the  existing  facilities,  and  to  broaden 
the  simulation,  stimulation,  and  command  and  control  capabilities. 

B .  PROGRAMMED  AND  APPROVED  CAPABILITIES 
1.  ACETEF9 

The  current  ACETEF  program  represents  cumulative  expenditures  of  about  $250 
million  through  FY  1992.  About  $80  million  in  additional  work  has  been  approved  and  is 
programmed  in  the  FYDP  to  bring  the  ACETEF  to  its  mature  configuration.  Table  3 
summarizes  the  funding  associated  with  the  program.  The  paragraphs  below  provide  a 
brief  narrative  of  the  approved  oihancements. 


Tabla  3.  ACETEF  Currant  and  Planned  Plant  Invastmant  Valua 


1989  and 

To 

Before 

1990 

1991 

1992 

1993 

1994 

Comnlete 

Military  Consiiuctioa 
(MDjCON) 

RF-sbieMed  hangar 

$86.3« 

Small  anedmic  diamber 

18.3« 

Closed  loop 

1.0 

Manned  flight  simulator 
Aircraft  systems  lab 

Night  combat  test  lab 

4.5 

12.3 

14.7 

Subtotal 

$110.1 

$12.3 

$0.0 

$0.0 

$0.0 

$14.7 

$0.0 

Labonaory 

Dired  prqiect  support 

$282 

$2.0 

$1.0 

$1.0 

$1.5 

$1.5 

$1.5 

l&M 

37.5 

3.6 

12 

1.1 

1.5 

1.5 

1.5 

CTEIP 

9.15 

9.6 

13.5 

15.1 

17.6 

23.0 

Other 

172 

1.0 

1.0 

Subtotal 

$82.9 

$15.75 

$12.8 

$15.6 

$18.1 

$20.6 

$26.5 

Total 

$193.0 

$28.05 

$12.8 

$15.6 

$18.1 

$35.3 

$26.5 

Cumulative  total 

$193.0 

$221.05 

$233.85 

$249.45 

$26755 

$302.85 

$329.35 

Note:  All  values  lepsesent  actual  past  or  planned  investment  in  millions  of  FY  1992  dollars,  unless  otherwise 
noted. 

■  Current  replacement  value. 


^  In  additioo  to  the  soaroe  previously  identified,  desaiptive  material  on  the  enhanormrnts  has  been  taken 
from  Reference  [10]. 


A  multi-player  environment  will  be  provided  by  completing  the  outer  ring  of  Hguie 
2.  The  Operations  and  Control  Center  will  provide  simulation  and  control  for  multi-player, 
multi-laboratory  test  scenarios  via  its  range  command  and  control  center  and  red  and  blue 
command  and  control  centers.  Up  to  IS  associate  players  are  provided  via  mini-crew 
stations.  Enhanced  software  capabilities  will  provide  improved  threat  generation  and 
control  known  as  Simulated  Warfare  Environment  Generator  (SWEG). 

The  Manned  Flight  Simulator  will  be  modified  to  enhance  visual,  motion,  and  aural 
cues  provided  to  aircrew  members  in  high-fidelity  cockpit  replicas  located  physically  away 
from  the  chamber,  but  integrated  with  the  system  under  test 

Modifications  to  EWISTL  will  provide  portable  and  millimeter-wave,  open-loop 
simulators,  jammer  simulators,  communications  environment  simulation,  additional 
background  threats,  and  missile  launch  simulation. 

An  Offensive  Sensors  Laboratory  (OSL)  will  provide  a  capability  to  generate  multi- 
spectral  targets  for  offensive  sensors  such  as  forward  looking  infrared  (FLIR),  infrared 
search  and  track  (IRST),  radar,  laser  sensors,  and  sensors  associated  with  smart 
munitions. 

The  NIFFTE  will  be  modified  to  a  communications,  navigation  and  identification 
(CNI)  laboratory  by  providing  RF  simulation  of  the  friendly,  foreign,  and  hostile  CNI 
environment;  and  to  (Hovide  a  CM  hardware-in-the-loop  capability. 

The  EMEGS  is  to  be  upgraded  to  the  E^TL  by  increasing  power  ouq)ut  and  adding 
emitters.  Additional  enhancements  will  improve  the  capabilities  of  the  Aircrew  Systems 
Evaluation  Facility  and  the  closed-loop  simulator. 

The  mature  ACETEF  will  represent  a  si^iificant  installed  systems  test  capability.  It 
is  not,  however,  without  limitations.  Although  a  second,  larger  anechoic  chamber  was 
under  consideration  at  one  time,  it  is  not  currently  part  of  an  approved,  funded  program. 
This  limits  use  of  ACETEF  to  tactical-sized  ainaaft,  and  generally  to  one  aircraft  at  a  time  in 
the  chamber.  Large  aircraft  testing  and  multi-aircraft  testing  are  not  possible  in  the 
chamber,  although  the  shielded  hangar  at  ACETEF  affords  a  limited  capability.  In 
addition,  the  principal  methods  of  stimulation  of  aircraft  systems  are  through  signal 
injection  downstream  of  the  antenna,  or  via  "antenna  hats"  that  mount  directly  on  the 
antenna.  Actual  radiation  of  signals  through  "free  space"  is  not  generally  possible,  at  least 
in  part  due  to  the  size  of  the  anechoic  chamber.  The  signiHcance  of  this  is  a  point  of 
contention  within  the  EC/avionics  test  community,  and  is  discussed  in  Appendix  G. 


25 


Finally,  the  closed-loop  capability  available  at  ACETEF  is  oriented  toward  sea- 
based  threats,  and  is  of  less  value  in  measuring  the  performance  of  systems  under  test 
against  land-based  threats.  In  addition,  from  the  perspective  of  the  system  under  test, 
threats  from  the  closed-loop  simulator  constitute  oidy  a  few  of  the  hundreds  of  threats 
generated  in  the  overall  test  scenario.  Notwithstanding  these  limitations,  ACETEF 
represents  the  only  signiricant  closed-loop  capability  currently  available  in  an  ISTF 
configuration,  and,  with  programmed  enhancements,  will  be  the  only  DoD  Category  I 
facility. 

2.  ECIT 

For  purposes  of  this  discussion,  we  have  treated  the  ECIT  as  having  no  currently 
approved  enhancements.  The  ECIT  was  included  in  the  FY  1992-97  FYDP,  which 
supports  the  FY1992  President's  Budget  with  approved  funding  of  $10  million  per  year  in 
T&E  investment  program  elements  for  a  total  of  $60  million  through  the  end  of  the  FYDP. 
The  Air  Force  provided  information  during  this  study  that  indicated  that  this  funding  paid 
for  only  limited  enhancements  to  the  ECIT,  and  that  additional  funding  by  customer 
programs  was  required  if  a  signiBcant  installed  systems  test  capability  were  to  be  provided. 
Congress  disapproved  the  $10  million  in  FY  1992  funds,  and  the  OfHce  of  the  Secretary  of 
Defense  (OSD)  removed  the  FY  1S)93  funds  from  the  FY  1993  President's  Budget  At  this 
time,  the  Air  Force  Program  Objective  Memorandum  (POM)  is  thought  to  contain  $10 
million  per  year  in  FY  1994-97,  pending  the  outcome  of  the  OSD  study  directed  by 
Congress.  A  detailed  discussion  of  the  ECIT  cost  and  funding  estimates  can  be  found  in 
Appendix  A.  Additional  enhancements  that  could  be  undertaken  at  ECIT  ate  discussed  in 
the  next  subsection,  but  to  our  knowledge  they  ate  not  part  of  an  approved,  funded  Air 
Force  program  at  this  time. 

C.  POSSIBLE  ENHANCEMENTS  (NOT  APPROVED) 

Previously  in  this  section,  we  mentioned  that  ACETEF  is  currently  limited  to 
tactical-sized  aircraft,  although  some  testing  of  less  stressing  test  scenarios  is  possible  in 
the  ACETEF  shielded  hangar.  This  precludes  the  testing  of  many  aircraft  (e.g.,  CV-22,  C- 
130,  P-3,  Joint  Stars,  E-2C,  E-6,  F/EF-111,  B-IB,  B-2)  and  tests  of  multiple  tactical 
aircraft  operating  simultaneously,  a  stated  Navy  and  Air  Force  requirement  In  addition, 
the  current  ACETEF  closed-loop  capability  does  not  cover  all  Air  Force  threat 
requirements.  The  current  ECIT  represents  a  very  limited  installed  system  test  capability. 
Enhancements,  beyond  those  that  ate  currmtly  approved,  ate  possible  at  either  or  both 
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locations.  An  additional,  larger  chamber  at  ACETEF  would  alleviate  some  of  the 
limitations  there;  and  also  would  present  opportunities  for  increasing  the  total  throughput 
{ot  cs^>acity)  of  ACETEF,  if  necessary  or  desired.  As  an  alternative,  improved  simulation, 
stimulation,  and  command  and  control  capabilities  could  be  added  to  ECIT  to  improve  its 
overall  cq>ability.  These  enhancements  are  described  below. 

1.  ACETEF 

The  Navy  has  considered  the  possibility  of  building  a  second,  large^°  anechoic 
chamber  at  ACETEF.  Although  no  definitive  design  had  been  selected  at  the  time  the 
pn^x)sal  was  shelved,  the  preliminary  design  called  for  a  chamber  that  would  be  200  feet 
(width)  by  185  feet  (length)  by  60  feet  (height).  This  would  provide  a  large  aircraft  test 
capability,  but  would  provide  little  capability  to  test  multi-aircraft  formations.  Also,  that 
size  of  chamber  would  not  support  free-space,  far-field  testing,  largely  limiting  tests  to  the 
Navy  approach  of  signal  injection  and  "antenna  hats."^^  The  additional  chamber  would 
permit  an  increase  in  throughput,  from  2.5  shifts  per  day  currently  possible,  to  about  4 
shifts  per  day.  Achieving  this  added  capacity  would  require  some  modest,  but  at  this  time 
undefined  additional  laboratory  capacity  to  stimulate  and  simulate  the  two  chambers.  The 
cost  of  a  second  chamber  and  the  degree  to  which  it  would  satisfy  additional  test 
requirements  ate  addressed  in  Section  VI  of  this  report 

Addititmal  closed-loop  simulation  capability,  more  representative  of  land-based 
threats,  will  be  needed  if  ACETEF  is  to  be  useful  for  EC  testing  of  Air  Force  systems. 
However,  this  requirement,  and  the  cost  of  satisfying  it,  is  common  to  both  ACETEF  and 
ECIT.  Consequently,  its  cost  was  not  further  considered  in  this  study. 

2.  ECIT  with  Additional  Simulation  and  Stimulation  Capabilities^^ 

The  ECIT  requites  additional  simulation,  stimulation  and  command  and  control 
caqMibility  to  be  an  installed  systems  test  facility.  The  AFFTC  briefed  the  study  team  on  the 
enhancements  that  had  been  under  consideration  in  earlier  ECIT  program  phases.  The  most 
capable  system  was  termed  the  "vision,"  a  level  of  capability  for  which  the  Air  Force  now 


If  we  use  the  termincdogy  of  the  Reliance  Report  [3],  the  chamber  most  recently  undm  serious 
considerarioii  by  the  Navy  would  Efdl  into  dm  "intennediaie”  toe  category.  However,  in  this  suidy,  we 
have  adopted  the  convent  of  "small”  and  "large"  only. 

I  ^  The  Air  Force  believes  that  at  least  some  free  space  radiation  testing  wiU  be  necessary  to  test 
EC/avioiiics  systems  in  the  future. 

Much  of  the  information  in  this  section  is  drawn  from  Reference  [23  through  25]. 
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says  there  is  no  requirement.  The  vision  enhancements  are  grouped  in  "clusters,"  as 
shown  in  Figure  4,  and  together  would  bring  ECIT  up  to  a  Category  1  level  facility. 
Funding  constraints  caused  the  program  to  be  limited  conceptually  to  creation  of  an 
improved  facility  infrastructure,  for  which  $60  million  was  approved  in  the  FYDP. 
Additional  enhancements  were  viewed  by  the  AF  as  "to  be  built,  when  and  if  needed." 
Although  not  explicitly  so  stated,  the  clear  implication  is  that  the  customer  program  with  the 
need  would  be  expected  to  finance  the  enhancements,  consistent  with  a  speciHed  ECIT 
architecture. 

The  basic  ($60  million)  infrastructure  enhancement  was  to  be  the  "anechoic 
cluster,"  depicted  in  Figure  S.  The  cluster  would  provide  standard  interfaces  between  a 
(new)  control  center  and  TEMS,  IFAST,  and  the  chamber.  It  also  would  provide  upgrades 
to  the  facility  operations  and  control  center,  would  improve  the  simulation  and  test  control 
capability  by  providing  better  scenario  generation  and  control;  and  would  enhance  the  RF 
threat-generation  capability.  Although  approved  funding  was  $60  million,  the  estimated 
cost  is  about  $72  million. 

To  describe  this  infrastructure  enhancement  in  greater  detail,  and  to  relate  it  to  the 
most  stringent,  known  test  requirement  (the  F-22),  it  is  necessary  to  depart  from  the 
"cluster"  description  of  the  Air  Force  ECIT  vision,  and  instead  refer  to  an  alternative 
"building  block"  approach,  also  used  by  the  Air  Force  to  define  potential  enhancements. 
The  building  block  approach  is  shown  in  Figure  6.  The  infrastructure  improvements  all 
affect  the  two  bottom  rows  and  are  unshaded.  The  capabilities  of  the  infrastructure  blocks 
are  [28]: 

•  Intrafacility  Communication — provides  the  communications  netwoik  within  the 
ECIT  for  digital,  video,  and  voice  signals. 

•  Test  Direction  and  Conduct — provides  the  hardware  and  software  for 
conducting  tests,  and  provides  consoles  and  stations  to  the  test  director  and  the 
test  conductor. 

•  Timing  and  Control — provides  synchronization  and  control  of  system 
computers  and  other  hardware  during  a  test 

•  Data  Collection  and  Analysis — provides  collection  of  data  from  the  system 
under  test  and  from  chamber  environment;  provides  for  display,  processing, 
and  storage  of  the  data;  collects  data  at  frequencies  from  DC  to  18  GHz  and  at 
rate  of  1  to  10  Mbits/sec. 

•  Engineering  Support — provides  test  development  system  support  tools. 
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Figure  4.  ECfT  Architecture:  The  "Vision 
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•  Perfonnance  Monitoring — ^provides  the  ability  to  monitor  the  performance  of 
the  ECIT  itself,  and  the  validity  of  the  test  semp. 

•  Test  Item  Support — ^provides  cooling  air,  hydraulics,  electrical  power,  etc.,  to 
the  system  under  test 

•  Fundamental  RF  Threats — provides  specific  threat  replica  stimulation's  to  the 
system  under  test  The  threats  provided  here  are  described  by  the  Air  Force  as 
those  needed  for  initial  F-22  testing  as  well  as  most  other  fighter  and  bomber 
tests,  and  include  surface-to-air,  air-to-air,  friendly,  hostile,  and  other  RF 
sources. 

•  High-Density  RF  Background — ^provides  the  overall  RF  density  representative 
of  the  theater  of  operation,  and  upon  which  the  fundamental  RF  threats 
outlined  above  are  superimposed.  Overall  objective  is  a  density  of  greater  than 
10  million  pulses  per  second. 

•  C^/C^I  Modeling — provides  the  command,  control  and  communications  for  the 
enviroiunent  of  the  fundamental  RF  threats  and  the  background  density 
representative  of  the  RF  background. 

The  above  c^bilities  are  all  within  tl»  $60  million  proposed  program.  However, 
the  F-22  minimum  facility  requirements,  based  on  information  provided  during  our  visit, 
are  the  three  bottom  rows  of  Figure  6,  designated  as  ATF  Category  2.  Using  the  building 
block  definitirais,  the  additional  capabilities  required  are: 

•  Customer  Plug-ins — provide  unique  RF  threats  and  targets  the  test  program 
desires  for  the  test  environment 

•  Specialized  Instrumentation — any  instrumentation  needed  to  satisfy  unusual 
requirements  of  the  test  custcaner. 

•  Customer  Communications — provides  special  links  for  customer  off-site  data 
networks  and  links  to  other  AFFTC  test  facilities. 

•  Radar  Cross-Section  R&M — provides  the  ability  to  assess  the  reliability  and 
maintainability  (R&M)  of  near-Held  radar  cross-section  (RCS)  of  the  test 
article. 

•  EO/IR  Target  Generator— provides  targets  for  offensive  sensors  operating  in 
the  EO/IR  frequency  bands. 

•  RF  Target  Generator — provides  RF  targets  for  the  radar. 

•  niase  measurement — provides  the  capability  to  test  phase  interferometer 
systems. 

There  is  no  easy  way  to  "mtq)"  these  blocks  to  either  the  clusters  shown  earlier  in 
the  ECrr  vision,  or  to  the  ciq>abilities  previously  discussed  in  the  Navy's  ACETEF.  In  a 
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general  sense,  the  additional  blocks  correlate  to  the  ACETEF  Offensive  Sensors 
Laboratory,  EWISTL,  CNI  laboratory,  Advani^  Flight  Simulator,  and  the  Operations  and 
Control  Center.  Not  provided  at  this  level  of  investment  is  either  the  ability  to  correlate 
multi^)ectral  threats  and  targets,  or  the  ability  to  link  with  other  DoD  test  facilities. 

The  net  result  of  all  the  potential  ECIT  enhancements  discussed  in  this  section 
would  be  a  Category  n  ISTF  capability  roughly  equal  to  that  expected  to  be  available  at 
ACETEF  upon  completion  of  iq>ptoved  ACETEF  upgrades,  except  as  follows: 

•  The  EOT  would  provide  a  large  chamber  capability. 

•  The  ECIT  threats,  communications  background,  and  targets  would  be  more 
"land-based"  than  "sea-based." 

•  There  would  not  be  a  closed-loop  capability. 

•  There  would  continue  to  exist  differences  in  philosophy  and  test  methods  in  the 
areas  of  signal  injection  versus  free-space,  far-Held  radiation.  Only  ECIT 
would  have  the  potential  for  both  free-space,  far-field  signal  radiation  and 
injection. 

The  estimated  cost  of  facility  upgrades  to  the  ECIT  are  [xesented  in  Section  VI. 
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V.  REQUIREMENTS 


The  purpose  of  this  section  is  to  discuss  the  requirements  for  testing  at  ACETEF 
and  ECTT  facilities.  It  starts  with  a  discussion  of  some  of  the  uncertainties  associated  with 
attempting  to  forecast  future  electrcmic  comb^  testing  needs,  and  the  risks  if  testing  is  not 
done.  This  is  followed  by  descriptions  and  tables  of  the  Navy  and  AF  test  requirement 
submissions.  The  final  subsection  discusses  how  IDA  used  this  information. 

A.  FORECASTING  DIFFICULTIES 

We  asked  the  Navy  and  the  Air  Force  to  project  their  anechoic  chamber/ISTF 
requirements  for  FY  1995  through  2005.  The  Navy  provided  forecasts  only  from  FY  1994 
through  FY  1996.  These  data  were  in  some  detail,  broken  out  by  requirements  for  the 
Aircraft  Anechoic  Test  Facility  (AATF),  a  small  anechoic  chamber,  and  for  a  large  anechoic 
chamber,  as  well  as  historical  usage  data  for  the  AATF,  but  no  information  for  the  other 
outyears  of  interest  The  Air  Force  projected  potential  average  annual  usage  over  the  next 
18  years  (FY  1992-2009)  of  electronic  combat  installed  systems  test  facilities,  by  aircraft 
program.  There  is  no  breakout  by  year  for  this  information. 

The  Major  Range  and  Test  Facility  Base  (MRTFB)  installations,  including  Patuxent 
River  NAS  and  Edwards  AFB,  routinely  project  future  requirements  for  their  facilities. 
The  installation  planners  use  specifics  gleatred  from  discussions  with  weapon  system 
program  offices  and  other  users.  They  generally  factor  up  the  program  office  estimates 
based  on  their  experience  with  specific  test  facility  usage.  They  do  so  because  program 
offices  often  do  not  confinn  expected  use  early  enough  for  planning,  and  because  program 
estimates  are  usually  low.  Two  principal  reasons  for  program  offices  making  low  usage 
estimates  are:  (1)  optimism  in  forecasting  uncertain  requirements  for  specific  tests,  and  (2) 
the  program  office's  perception  that  it  may  be  forced  to  buy  a  test  capability  that  it  needs  but 
which  is  not  currently  available. 

As  to  the  first,  program  offices  manage  tlreir  programs  to  budgets  that  are  sensitive 
to  program  schedule.  The  goal  of  the  program  is  to  get  in  and  out  of  testing  in  the 
minimum  time  consisrent  with  achieving  tte  test  goals.  This  optimism  may  cause  an 
underestimate  of  the  time  in  test  Each  new  program  expects  to  have  solved,  prior  to 
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testing,  problems  that  plagued  earlier  programs.  But  new,  unforeseen  difficulties  often 
arise  that  require  other  testing.  Program  offices  do  not  want  to  lose  flexibility  by  tying  their 
programs  to  specific  test  scenarios  that  may  or  may  not  come  to  pass. 

Secondly,  program  offices  usually  fund  program-specific  facility  requirements  on 
test  installations.  [For  generic  capabilities,  the  installation  initiates  its  own  requirements 
through  Improvement  and  Modernization  (I&M)  and  Military  Construction  (MILCON) 
funding.  Requirements  for  multi-Service  national  test  assets  may  be  met  through  the  OSD- 
administered  Central  Test  and  Evaluation  Investment  Program  (CTEIP)].  Stating  a  need 
for  an  unavailable  test  capability  may  result  in  the  program  office  being  required  to  fund  the 
facility  that  fvovides  it 

Complicating  the  problem  for  independent  analysis  is  that  many  of  the  systems  of 
interest  in  electronic  combat/avionics  test  programs  are  classified.  Thus,  we  had  little 
opportunity  to  independently  check  the  reasonableness  of  Service  test  requirements 
estimates.  Instead,  we  compared  future  projections  with  needs  met  and  unmet  by  the  same 
or  similar  aircraft  programs  in  the  past  These  analogies  gave  us  confidence  that  the  future 
needs  were  reasonable. 

Moreover,  the  complexity  of  avionics  and  enhancements  to  test  capability  are 
evolving  rapidly.  Control  of  avionics  processing  is  changing  from  "federated"  to 
"federated  with  communication"  to  partially  (and,  eventually,  fully)  "integrated"  systems 
(see  Section  HI).  No  one  yet  knows  how  to  test,  much  less  how  much  testing  will  be 
required  for  a  completely  integrated  EC/avionics  system.  (Because  of  this  increasing 
interconnectivity  and  integration,  we  use  the  convention  "EC/avionics"  to  describe  the 
installed  systems  being  tested  in  the  anechoic  chamber.) 

B.  RISKS 

In  their  discussions  with  IDA,  both  facility  and  program  office  personnel  made  it 
clear  that  there  are  always  tradeoffs  between  levels  of  testing,  availability  of  satisfactory  test 
facilities,  cost,  and  schedule  on  one  hand,  and  risk  to  mission  performance  and  safety  on 
the  other.  Some  things  cannot  be  tested  because  there  is  no  acceptable  way  to  test  them. 
Others  are  not  tested  to  the  fullest  because  it  is  simply  too  costly  for  the  marginal  value 
received.  Program  managers  and  testos  routinely  make  judgments  as  to  how  much  ride 
they  are  willing  to  accept  to  keep  a  program  on  cost  and  schedule.  In  that  sense,  test 
"requirements"  should  not  be  considered  absolute.  Still,  it  is  usually  more  expensive  to 
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correct  installed  avionics  errors  in  production  than  to  discover  and  correct  the  error  during 
development 

C.  NAVY  REQUIREMENTS  FOR  ACETEF 

Figure  7  shows  the  Navy  projections  for  FY  1994  through  FY  1996  for  Navy 
systems  expected  to  use  the  capabilities  of  ACETEF  through  either  the  Shielded  Hangar, 
the  current  small  anechoic  chamber  (SAC),  or  a  postulated  large  anechoic  chamber  (LAC) 
at  Patuxent  River  NAS.  It  does  not  include  anticipated  usage  by  the  Army,  Coast  Guard, 
Air  Force,  or  foreign  customers.  The  Army's  RH-66  helicopter  could  become  a  major 
customer  in  the  future  because  it  has  integrated  avionics  and  is  expected  to  be  tested  at 
ACETEF. 


Fncal  Year 

O  Shielded  Hangar  H  Anechoic  Chamber  B  Larger  Anechoic  Chamber 

Figure  7.  Navy  Chamber  Requirements 

The  shielded  hangar  at  Patuxent  NAS  allows  testing  of  more  than  one  aircraft 
simultaneously,  and  a  review  of  historical  data  since  calendar  year  1987  shows  that  an 
average  of  three  to  four  different  aircraft  are  tested  each  week  there.  Based  on  that  data  and 
those  for  the  AATF,  Navy  usage  can  be  expected  to  average  more  than  one  shift  per 
workday  for  all  three  categories  of  chambers.  Because  personnel  at  Patuxent  tell  us  that  a 
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second  shift  (or  even  2.5)  is  feasible  for  their  anechoic  chamber  now,  we  assume  that  there 
is  unused  opacity  at  ACETEF  now  and  in  the  foreseeable  future. 

Table  4  shows  specific  systems  test  requirements  for  chambers  utilizing  ACETEF, 
as  provided  by  the  Navy.  The  rows  reflect  the  aircraft  platforms  and,  in  some  cases, 
specific  EC/avionics  systems  under  test  The  columns  show  requirements  by  fiscal  year 
broken  out  by  expected  use  of  the  shielded  hangar  (SH)  or  the  SAC,  both  of  which  are  on- 
hand,  or  a  projected  LAC.  (The  last  is  not  as  large  as  the  BAF  at  the  AFFTC,  so  it  is 
sometimes  called  the  intermediate  anechoic  chamber.)  The  Navy  input  was  in  workdays. 
IDA  converted  this  to  workweeks  (5  workdays  per  woficweek)  to  be  consistent  with  the  Air 
Force  submission  discussed  in  the  next  section. 

Items  in  Table  4  under  "GTRI”  ate  from  Georgia  Tech  Requirements  Inventory  [4] 
and  were  included  in  the  requirements  used  to  project  Navy  demand  in  Project  Reliance 
deliberations. 

Items  under  "NATC  Probable"  were  developed  by  the  Naval  Air  Test  Center 
(NATO,  now  the  Naval  Air  Warfare  Center,  Aircraft  Division  (NAWC-AD).  This 
computation  was  based  on  the  System  Engineering  and  Test  Directorate's  assessment  of 
anticipated  usage,  and  was  approved  and/or  modified  by  the  i»ogram  managers  (PMs).  As 
indicated  in  the  table,  some  of  the  systems  were  changed  by  the  PMs  and  some  have  since 
been  canceled  by  the  Navy.  We  retained  tteir  stated  needs  as  "place  holders"  for  future 
pre-plarmed  product  improvement  (P^I)  of  systems  they  were  meant  to  replace,  or  of  other 
replacement  systems  that  will  perform  similar  functions.  At  the  bottom  of  the  table, 
"Shifts"  reduce  the  average  weeks  required  in  the  small  and  large  anechoic  chambers  to  the 
number  of  equivalent  work  shifts  required,  assuming  48  weeks  per  year. 

As  already  stated  the  Navy  requirements  do  not  include  specific  Army,  Coast 
Guard,  or  Air  Force  systems  which  also  use  ACETEF.  During  FY  1989  through  FY  1991 
these  requirements  represented  only  about  6%  of  the  NAWC-AD  chamber  workload. 
Navy  requirements  for  the  shielded  hangar  ate  included  in  the  table,  but  do  not  enter  into 
the  analysis  of  requirements  for  anechoic  chambers  presented  in  later  sections,  because  the 
shielded  hangar  is  not  a  substitute  for  an  anechoic  chamber. 

D.  AIR  FORCE  REQUIREMENTS  FOR  ISTFS 

The  Air  Force  Electronic  Combat  Office  (AFECO)  at  Wright-Patterson  AFB  was 
the  focal  point  for  Air  Force  requirements  for  use  of  installed  systems  test  facilities 
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Tabto  4.  Navy  Eatimata  of  Chambar  Raquiramanla  (Waaka  Par  Yaar) 
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*  Cancded  Other  F^I  foUow-on  systems  expected  to  genenie  simUv  tequiiemeitts. 
!>  NATC  projection  modified  by  prognm  manageis. 


(ISTF)  for  the  Project  Reliance  deliberations.  (Project  Reliance  is  an  ongoing  initiative 
within  the  T&E  community  to  assign  Service  and  Defense  Agency  leads  to  functional  areas 
of  testing;  these  leads  have  a  duty  to  minimize  test  facility  duplication  through  a  number  of 
procedures  such  as  needs  validation  and  recommending  centers  of  excellence  to  receive 
available  funding  for  improvements.)  The  AF  requirements  are  shown  in  Table  5  and  total 
190  weeks  per  year. 


Table  5.  Air  Force  Estimate  of  ISTF  Requirements 
(Weeks  Per  Year,  18-Year  Average) 


Proarams 

Reliance 

Estimate 

Current 

Estimate 

B-ia 

18 

3 

B-2a 

0 

2 

c-na 

2 

1 

CV-22 

0 

12 

C-130a 

2 

1 

AC-130Ua 

0 

6 

EC-130a 

18 

20 

MC-130CTa 

16 

12 

C-14la 

2 

1 

E-3a 

4 

5 

JSTARS® 

6 

5 

F-15 

16 

11 

F-16 

16 

10 

F-22a 

27 

10 

EF-111 

18 

19 

F-111 

14 

9 

B-52a 

9 

0 

TR-l/U-2 

8 

0 

RC-135a 

14 

0 

Total  AF 

190 

127'’ 

B  Requires  a  large  anecboic  chamber, 
b  With  20%  scheduling  factor,  this  total 
becomes  152  weeks  per  year. 


To  answer  the  current  tasking,  AFECO  convened  an  ISTF  Requirements 
Conference  of  program  offices,  operational  test  organizations  and  AF  major  air  commands 
(the  users)  to  review  the  Reliance  projections  for  relevancy  to  the  FY  1995-2005  time.  The 
negotiated  final  result,  the  average  of  requirements  for  ISTF  over  the  18-year  period,  is  as 
shown  in  the  last  column  of  Table  5.  The  total  need  fell  33%  from  190  to  127  weeks  per 
year.  However,  AFECO  believes  that  a  20%  longer  schedule  is  needed  to  ensure  that  the 
127  weeks  per  year  in  the  chamber  would  actually  be  met.  The  Navy  point  of  contact 
agreed  that  this  was  a  good  planning  factor.  This  20%  raises  the  AF  requirement  to  152 
weeks  per  year.  (This  20%  was  added  to  both  Navy  and  Air  Force  requirements.  IDA 
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performed  sensitivity  analyses  both  with  and  without  the  20%  factor,  for  Navy  as  well  as 
Air  Force  requirements.) 

A  total  of  51  weeks  (40%)  of  the  127  weeks  per  year  AF  aimual  requirement  are  for 
systems  too  large  to  fit  into  the  AATF  at  NAWC-AD.  In  order  to  be  adequately  tested, 
these  platforms  will  require  a  facility  such  as  the  existing  Benefield  Anechoic  Facility  at 
Edwards  or  a  new  large  anechoic  chamber  at  NAWC.  An  as  yet  unmet  requirement 
expressed  by  several  Navy  and  AF  tactical  aircraft  programs  is  that  of  testing 
electromagnetic  compatibility  among  multiple  aircraft  in  the  same  chamber.  This  also  can 
be  dcme  only  in  a  chamber  larger  than  the  AATF. 

The  Air  Force  requirements  for  installed  systems  test  facilities  is  shown  with  Navy 
actuals  and  requirements  in  Figure  8.  The  breakout  of  AF  small  and  large  chamber 
requiremotts  was  done  by  IDA  based  only  on  the  size  of  aircraft. 


□  1988  □  1988  H  1990  B  1991 

I  B  RequirviTMnts  0  Reliance 

in  This  Study 

Note:  Data  for  1988  through  1991  are  actual. 

Figure  •.  Navy  and  Air  Force  ISTF  Raquiramante 
(Comparieon  of  Waake  par  Year) 
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The  scale  is  in  average  number  of  shifts  per  day  over  the  course  of  a  year.  One 
shift-year  equals  8  hours  per  day,  48  weeks  per  year  of  anechoic  chamber  usage  on 
average.  The  remaining  four  weeks  per  year  is  downtime  for  maintenance,  upgrades, 
holidays,  etc.  Navy  actuals  are  of  utilization  of  the  small  anechoic  chamber  at  ACETEF 
("Navy,  Small")  for  FY  1988  through  FY  1991.  We  see  in  the  figure  that  Navy  usage  has 
grown  to  one  shift  equivalent  through  FY  1991,  with  the  planned  MTP  (Manage  to 
Program)  equal  to  1.5  shifts  in  FY  1992.  Service  forecasts  of  ISTF  needs  (Requirements 
in  Study)  in  almost  all  cases  have  been  validated  by  the  applicable  program  management 
offices.  Validated  (and  unconstrained)  Navy  projections  are  for  two  shift-year  equivalents 
over  FY  1994  through  FY  1996.  The  other  series  in  the  figure  show  forecasts  of  AF  small 
chamber  and  Navy  and  AF  large  chamber  requirements.  AF  forecasts  for  both  tactical  and 
large  aircraft  are  lower  in  total  than  their  Navy  counterparts.  In  summary,  the  Service 
forecasts  of  ISTF  requirements  for  electronic  combat/avionics  ground  testing  seem  in 
consonance  with  past  Navy  experience. 

If  one  compares  the  AF  requirements  with  those  of  the  Navy  for  somewhat  similar 
systems,  it  would  appear  that  either  those  of  the  Air  Force  are  understated  or  the  Navy's  are 
overstated.  We  believe  that  those  of  the  Air  Force  are  low,  periiaps  because  some  in  the 
AF  do  not  have  a  full  aj^reciation  of  such  capability  as  has  been  developed  by  the  Navy  at 
ACETEF.  Aircraft  systmis  with  no  stated  requirements  include  the  C-5,  the  C-27,  and  the 
F-117. 

E.  INSTITUTE  FOR  DEFENSE  ANALYSES  ESTIMATES 
1 .  Assumptions 

We  assumed  for  the  purpose  of  our  analyses  that  the  Navy  requirements  data  reflect 
annual  requirements  not  only  through  FY  1996  but  also  for  the  foreseeable  future.  We  also 
assumed  that  reductions  in  test  requirements  arising  from  the  ongoing  force  drawdown  and 
diminished  procurement  will  be  offset  by  increases  from  retrofits,  greater  avionics 
complexity,  and  expanded  testing  resulting  from  the  new  acquisition  policy  of  longer 
development,  thorough  evaluation,  and  deferred  production.  (This  also  could  mean  the 
testing  of  major  upgrades  to  existing  systems  at  the  same  time  as  new  (potentially 
replacement)  systrans  are  being  tested  in  engineering  develt^ment) 

Air  Force  and  Navy  experience  and  future  requirements  are  compared  in  Table  6. 
The  Air  Force  estimates  are  comparable  to  the  Navy  experience,  but  less  than  the  Navy 
estimates.  The  Air  Force's  current  estimate  for  tactical  aircraft  is  ^proximately  three- 
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fourths  of  the  Navy's  FY  1994-96  SAC  requirements.  The  Air  Force's  currmt  estimate  for 
large  aircraft  is  about  the  same  as  the  Navy's  LAC  requiremmits  for  FY  1994-96.  The 
F-IS  and  F-16  estimates  approximate  the  F-14  and  F-18  experience  and  are  less  than  the 
estimated  requirements  for  tte  F-14  and  F-18.  An  average  of  two  to  diree  wedcs  per  year 
in  an  anechoic  chamber  for  the  B-2  and  B-IB  and  none  for  the  F-1 17  is  questionable  when 
compared  with  other  systems.  On  the  whole,  the  Air  Force  requirements  appear  to  be 
cmservative  when  ctxnpared  to  Navy  experience. 


Tabla  6.  Waaka  par  Yaar  of  Taat  Raquiramanta: 
Air  Forca  Eatlmata  and  Navy  Exparianea 


Air  Force 
lYocnms 

Reliance 

Estimaie 

Qifient 

Estimaie 

NawBronams 

Requiiemenis 

Met 

RequiremeDts 

Unmet 

FY  1994-96 
Estimate 

Tactical 

91 

59 

SAC 

44 

134 

80 

F-15 

16 

11 

F-14 

9 

16 

17 

F-16 

16 

10 

F-18 

13 

39 

17 

F-22 

27 

10 

A-12 

11 

F-lll 

14 

9 

A-6 

2 

5 

4 

F-in 

V-22 

5 

13 

EF-111 

18 

19 

Miac. 

9 

28 

10 

Hek> 

Helo 

5 

22 

6 

Large 

99 

68 

LAC 

74 

B-IB 

18 

3 

P-7 

5 

B-2 

2 

B-52 

9 

JSTARS 

6 

5 

EP-3  ARIES 

4 

£-3 

4 

5 

E-2 

1 

2 

3 

Miac.* 

62 

53 

Miac.* 

62 

■  Specific  lyitenu  are  identified  in  Tables  4  and  S. 


We  had  to  assume  that  the  three  years  of  stated  Navy  requirements  are 
representative  of  those  throughout  the  period  of  interest,  and  that  the  18-year  annual 
average  of  AF  tests  requiring  an  ISTF  could  be  iq>plied  uniformly  in  all  years.  In  fact, 
there  would  be  peaks  and  valleys  of  demand,  but  these  could  normally  be  anoothed 
through  priorities,  and  we  so  assumed.  The  one  system  that  we  treated  differently  was  the 
F-22.  The  bulk  of  its  testing  is  expected  to  be  required  in  FY  1995-1999.  Because  of  the 
Systems  Program  Office  (SPO)  estimaies  of  sizable  deployment  costs,  we  had  to  treat  it  as 
a  special  case,  with  different  values  for  needs  met  before  FY  2000  and  after. 

2.  Requirements  for  Laboratories/Clusters  Supporting 
the  Anechtrfc  Chamber 

As  described  more  fully  in  Section  IV,  an  ACETEF  or  ECIT  capability  is  much 
more  than  a  mme  anechoic  chamber.  These  capabilities  represent  a  group  of  test  functicms 
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that  use  the  chamber  as  a  tool.  (The  Navy  uses  the  term  "laboratory"  and  the  AF  the  term 
"cluster"  to  identify  a  function  that  supports  testing  in  its  anechoic  chamber.  For 
consistency,  we  refer  to  them  as  "functions.")  The  data  provided  by  the  Navy  and  the  AF 
did  not  identify  the  functions  that  would  be  used  during  the  weeks  the  programs  were  in  the 
chamber.  (This  would  be  a  large  task  as  it  would  require  access  to  often  classified, 
sometimes  compartmented  material  detailed  in  specific  avionics  test  plans,  if  they  exist.) 
Members  of  the  Operational  Evaluation  Division  of  IDA  looked  at  the  list  of  AF  systems 
and  categorized  what  EC  test  functions  they  might  require.  This  was  based  on  an  informed 
judgment  of  the  platform's  intended  mission  and  its  anticipated  EC/:  vionics  suite. 

3.  Total  Electronic  Combat  Test  Requirements  for  ISTFs 

Figure  9  compares  historic  requirements  for  use  of  the  AATF  at  Patuxent  NAS  with 
those  forecast  by  the  Navy  and  Air  Force.  A  review  of  macro  data  on  use  of  the  AATF  in 
calendar  years  1989  through  1991  shows  weeks  in  the  chamber  of  40,  45,  and  48 
respectively,  or  increasing  to  1.0  shifts  on  a  48:1  basis.  During  those  years,  the  same  data 
indicate  that  NATC  turned  away  an  average  of  another  134  weeks  (three  shifts)  annually. 
(This  number  may  be  overstated  because  there  may  be  some  multiple  counting  of 
tumaways  requesting  the  same  testing  more  than  once,  and  of  tumaways  whose  requests 
were  later  filled.) 

The  AATF  is  expecting  to  use  1.5  shifts  in  FY  1992,  which  equates  to  72  weeks  in 
the  chamber.  This  is  labeled  MTP  (Manage  to  Program)  meaning  that  it  is  based  on  being 
able  to  serve  that  number  within  expected  funding,  etc.  The  AATF  FY  1994-1996  forecast 
of  small  chamber  requirements,  which  we  assumed  would  hold  constant  for  20  years,  is 
for  80  weeks  annually  (96  weeks  or  2.0  shifts  with  the  20%  schedule  flex).  Although 
50%  higher  than  the  current  level,  the  requirement  seems  modest  compared  to  the  sum  of 
historical  usage  plus  tumaways. 

We  do  not  have  any  AF  historical  data  but  the  AF  forecasts  look  roughly  right  when 
compared  to  those  of  the  Navy.  No  historical  data  are  shown  for  the  Navy  LAC 
requirements  either,  because,  except  for  some  B-IB  testing  and  perhaps  a  small  amount  of 
other  EC  testing  in  the  BAF,  a  significant  ISTF  capability  incorporating  a  LAC  does  not 
exist.  The  Project  Reliance  levels  are  shown  to  depict  the  AF  "scrub"  of  requirements  to 
get  to  the  levels  used  in  this  study.  Navy  LAC  requirements  include  tactical-sized  aircraft 
where  such  was  indicated  (see  Table  4).  All  of  the  F-22  requirement  but  no  other  tactical- 
size  AF  aircraft  are  shown  in  the "  AF,  Large"  requirements  because  we  had  no  break-out  of 
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Nunnbar  ol  Shifts  psr  Day 


the  F-22  needs  and  no  indication  that  other  AF  tactical  platforms  had  a  large  anechoic 
chamber  requirement,  although  some  (Hrobably  do. 


Nawy,  Small 
□  1988 


Wkh  Air  Fores.  SmaH  Navy,  Largs  Air  Fores,  Largs 
Tumawaya 

□  1989  P  1990  HI  1991 


I  1992  H  Rsquirsmanta  0  Rsianoa 

in  This  Study 


Nets:  Data  for  1988tirough  1991  arsacluaL 


Figure  9.  Electronic  Combat  ISTF  Requiremente 
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VI.  COST  AND  EFFECTIVENESS  ANALYSIS 


Previous  sections  of  this  report  covered  existing  ISTF  capabilities  and  possible 
enhancements  to  those  facilities  (Section  IV)  and  the  projected  Service  requirements  for 
ISTF  testing  (Section  V).  This  section  focuses  on  the  costs  and  effectiveness  of  the  ISTF 
alternative  enhancements  in  satisfying  the  requiranents. 

A.  COSTS 

In  this  subsection,  we  present  the  costs  associated  with  the  alternatives  previously 
identified.  Table  7  summarizes  the  cost  estimate  for  the  “base  case,"^^  and  includes 
investment,  20-year  operating,  and  deployment  costs. 


Table  7.  Cost  Summary 
(Milliona  of  Constant  FY  1992  Dollars) 


Cost  Element 

Alternative  Number 

1 

2 

3 

4 

Baseline  ACb'lEb 

$60 

$60 

$60 

hivesiment 

$0 

$68 

$168 

$236 

20-Year  Opeiating 

$0 

$54 

$229 

$283 

Deployineat  Cost 

$202 

$272 

$28 

$8 

Total 

$262 

$454 

$486 

$589 

Discounted  at  10%  per  year 

$167 

$249 

$264 

$328 

The  base  case  methodology  is  described  in  succeeding  paragraphs,  while 
excursions  to  the  base  case  ate  documented  in  Appendix  E.  AdditicHud  supporting  data  can 
be  found  as  follows: 

•  ^pendix  A  contains  data  on  investment  costs. 

•  Appendix  B  shows  the  methodology  for  converting  to  constant  FY  1992 
dollars. 

•  Appendix  C  contains  data  on  operating  costs. 

•  ^>pendix  D  cmtains  data  on  deployment  costs. 


The  base  case  uses  the  IDA  esdnuie  of  cost  of  dqA)yiiieat,  described  in  Section  VI.A3. 
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1 .  Investment  Costs 

We  previously  described  approved  and  programmed  enhancements  to  the  ACETEF. 
All  four  alternatives  we  considered  in  this  study  assume  that  these  enhancements,  described 
in  Table  7  as  the  Baseline  ACETEF,  will  be  completed,  at  a  cost  of  $60.5  million  in 
constant  FY  1992  dollars.  Tables  8  and  9  summarize  overall  investment  costs  associated 
with  all  four  alternatives.  Table  8  shows  investment  costs  in  constant  FY  1992  dollars  for 
all  ACETEF  capabilities.  While  future  costs  are  the  only  relevant  costs  for  investment 
decisions,  we  included  the  historical  (sunk)  costs  of  $271.5  million  as  well  as  the  future 
costs  of  $60.5  million  to  provide  a  complete  baseline  from  which  to  estimate  future  ECIT 
capabilities  and  related  costs  by  analogy.  Only  future  costs  for  ACETEF  are  included  in  the 
four  alternatives. 


Table  8.  ACETEF  Investment  Costs 
(Millions  of  Constant  FY1992  Dollars) 

Historical  Costs 
(Through  FY  92) 

Future  Costs 

Total 

Costs 

Baseline  ACETEF 

Building  Construction  (MILCON) 

Small  Anechoic  Chamber  (SAC) 

$18.3 

$18.3 

Shielded  Hangar  (SH) 

86.3 

86.3 

Aircrew  Systems  Evaluation  Facility  (ASEF) 

13.3 

13.3 

Lab  improvements  (RDT&E) 

EWISTL 

17.3 

$10.6 

28.0 

Manned  Flight  Simulator  (MFS) 

57.7 

9.4 

67.1 

Electromagnetic  EnvirxMunental  Effects  Test  Lab 

30.3 

3.5 

33.7 

Shielded  Hangar  (SH) 

Closed-Loop  Facility 

25.6 

3.7 

29.3 

Aircrew  Systems  Evaluation  Facility  (ASEF) 

3.1 

2.6 

5.7 

Operations  and  Control  Center  (OCC) 

4.2 

8.2 

12.4 

Offensive  Sensors  Lab  (OSL) 

5.8 

15.0 

20.8 

CNILab 

_Lfi 

17.2 

Total 

$271.5 

$60.5 

$332.0 

Large  Anechoic  Chamber  (LAC)  at  AtTETEF 

Building  Coastruction  (MILCON) 

$68.0 

Alternative  1  has  no  additional  investment  cost,  because  this  alternative  involves 
increased  throughput,  achieved  by  increasing  to  2.5  shifts  per  day.  No  additional  facilities 
are  required. 

Alternative  2  builds  a  large  anechoic  chamber  at  ACETEF,  and  costs  $68.0  million 
in  constant  FY  1992  dollars,  also  shown  in  Table  8.  The  cost  estimate  was  provided  by  the 
Navy,  who  had  proposed  to  build  the  chamber  several  years  ago,  and  studied  the  cost  in 
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some  detail.  We  accepted  the  value  as  reasonable.  By  comparison,  the  BAF  cost  about 
$50  million  when  it  was  built  as  part  of  the  B>1B  contract  in  1988. 


Table  9.  ECIT  Invastmant  Costs 
(Constant  FY1992  Dollara  in  Millions) 


Equivalent 

ACETEF 

Canabilitv 

Future  Costs 

EOT 

Geoeric  (Rsali^  Infiastnictufe) 

Anecfaoic  dumber  duster 

EWISTL 

28.0 

Management,  control 

OCC 

10.5 

Other 

Multiple 

14.8 

Program  management 

Multiple 

18A 

Total  generic  program 

71.9 

Full  Function  Category  2  ISTF 

Building  consttuction  (MILCON) 

Aucr^  Systems  Evaluation  Facility 

Lab  construction  (RDT&E) 

ASL 

13.3 

MFS  duster 

72.8 

CNI  cluster 

MFS+ASEF 

17.2 

ElectrcHiptiGal/Lafraied  duster 

CNI 

20.8 

RF/Electranic  wartare  duster 

OSL 

Management  and  comrol 

Multiple 

1.9 

Adjustments 

OCC 

Double  counting  of  equ^ment  interfaces 

-14.8 

Antkipaied  price  reductiaa* 

iMJ 

Total  long-term  program 

96.5 

Total  generic  idus  long-term  urograms 

168.4 

*  lS%ofhiitoricalc(MtofEWISTL.MFS,  ASL.OCC.OSL,andCNL 

Alternative  3  develops  the  ECIT  into  a  full  function  Category  2  ISTF  by  adding 
considerable  simulation,  stimulation,  and  command  and  control  capability.  We  broke  the 
cost  into  two  parts,  the  generic  infrastructure  c^)ability,  which  theoredcaUy  represents  what 
the  Air  Force  intended  to  provide  with  the  $60  million  program  that  was  to  start  in  the  FY 
1992  President's  Budget  and  the  additional  enhancements  needed  to  satisfy  minimum  F-22 
requirements.  (See  Section  TV.C.2.  for  a  discussion  of  these  enhancements.)  Table  9 
shows  all  future  investment  costs  in  FY  1992  constant  dollars  for  the  planned  ECIT 
capabilities.  We  did  not  include  historical  costs  because  they  were  not  relevant  to  the 
analysis.  As  shown  in  Table  9,  the  total  cost  is  $168.4  million.  The  generic  upgrade 
program  is  $71.9  million.  We  estimated  that  an  additional  $96.5  millicm  would  be  required 
to  meet  F-22  needs  and  would  periiaps  be  paid  from  program  funds.  Both  figures  were 
estimated  by  analogy  to  the  ACEi'EF  capabilities.  While  this  is  just  one  of  a  full  spectrum 
of  possible  ECIT  configurations  and  costs,  we  selected  the  configuration  above  because  it 
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gives  an  equivalent  capability  to  that  of  ACETEF.  The  $60  million  program  is  not 
suf&cient  in  itself. 

2 .  Operating  Costs 

For  each  alternative,  we  computed  operating  costs  that  were  additive  to  current 
operating  costs  fm*  ACETEF  and  ECTT;  that  is,  we  assumed  that  current  costs  will  continue 
and  do  not  vary  among  the  four  alternatives.  We  also  assumed  that,  apart  from  deployment 
costs,  which  are  treated  in  the  next  subsection,  the  cost  of  performing  any  given  test 
program  is  the  same  at  ECIT  or  ACETEF.  Consequently,  only  those  operating  costs 
associated  with  increased  facilities  (Alternatives  2, 3,  and  4)  have  a  bearing  on  the  analysis. 
We  calculated  the  20-year  additive  operating  cost  of  Alternative  2  at  $54.1  million. 
Alternative  3  at  $229.1  million,  and  Alternative  4  at  $283.2  million.  These  additive  costs 
are  due  to  increases  in  test  workload  supported  and  the  operation  of  an  additional  chamber 
in  Alternatives  2  and  4.  The  methodology  we  used  is  described  in  detail  in  Appendix  C. 
The  ECTT  operating  cost  appears  to  be  unusually  high,  but  that  is  because  the  supporting 
functions  are  low,  whereas  ACETEF  starts  from  its  present  high  capability.  The  operating 
cost  of  program-speciHc  testing  is  not  included.  We  assumed  it  would  be  q)proximately 
the  same  at  either  locaticm. 

3 .  Deployment  Costs 

The  Air  Force  position  is  diat  ground  testing  an  F-22  at  Patuxent  NAS  would  incur 
substantial  costs  over  and  above  those  for  testing  the  same  aircraft  at  Edwards.  The  reason 
is  the  F-22's  new  start  situation  and  its  integrated  electronic  combat/avionics  systems.  It 
also  is  tied  to  the  Air  Force's  testing  philosophy,  which  calls  for  collocation  due  to  close 
coordination  and  correlation  between  flight  and  ground  (chamber)  testing  (see  Section  m). 
At  the  same  time  that  F-22  Combined  Test  Force  (CTF)  personnel  and  equipment  would  be 
deployed  to  Patuxent  NAS,  a  full  complement  of  the  CTF  technical  persoimel  and 
equipment  would  have  to  be  left  at  Edwards  to  work  with  the  flight  test  program,  which 
would  remain  at  Edwards.  An  F-22  systems  integration  laboratory  (SIL)  with  two  avionics 
lines  would  be  constructed  at  Patuxent  NAS  with  75  people  to  run  it  Moreover,  the  Air 
F(»ce  states  that  results  obtained  in  Maryland  would  have  to  be  correlated  with  those  at  the 
IFAST  in  California,  so  there  would  be  no  savings  in  the  ground  test  program  at  the  latter 
location.  Not  only  would  the  avionics  test  aircraft  remaining  at  Edwards  be  unable  to  make 
up  the  flight  tests  now  scheduled  for  the  test  article  going  to  ACETEF,  but  25%  more 
additional  flights  would  be  requited  to  correlate  results  obtained  at  the  two  locations. 
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Hie  F-22  is  the  only  program  for  which  an  estimate  of  deployment  costs  to 
ACETEF  was  provided.  See  Appendix  F  for  the  deuul  and  rationale  of  this  estimate.  For 
all  the  other  Air  Force  programs  (and  for  Navy  programs  deployed  to  ECIT  for  some 
excursions  in  Alternative  4),  we  used  AF  standard  average  deployment  costs  for  tactical 
and  for  large  aircraft  systems.  AF  systems  such  as  the  F-16  and  the  C-130  have  deployed 
normally  to  ACETEF  for  short  periods  in  the  past,  but  none  was  an  extensive  new 
development  such  as  the  F-22. 

The  numbers  in  Table  10  show  the  deployment  costs  and  number  of  weeks  of 
testing  for  an  Air  Force  average  annual  testing  program  if  all  aircraft  were  deployed  for  the 
average  length  of  time  stated  in  the  requirements.  We  divided  the  deployment  costs  by  the 
testing  periods  because  the  costs  are  dependent  on  time.  Much  of  the  deployment  cost  is 
personnel  per  diem,  and  personnel  on  temporary  duty  for  testing  projects  exceeding  six 
weeks  in  duration  are  allowed  monthly  visits  home.  The  F-22  was  treated  as  a  special  case 
since  we  had  additional  information  from  the  program  office. 


Table  10.  Taating  Raquiramants  and  Daploymant  Costa 

ISTFTesting  Weeks  per  Year 

Including  20%  Costper  Year,  if 
Sdnduling  Depli^ed  (Millions 


Service  Data 

Allowance 

of  FY  1992  Dollars! 

Navy 

Tactical  aticraft 

80 

96.0 

Large  aiicraft 

74 

88.4 

Air  Force 

F-22 

Firsts  years 

21 

25.2 

Special  Case 

Thereafter 

8 

9.6 

Other  tactical  ancraft 

EF-111 

19 

22.8 

1.3 

F-15 

11 

132 

0.8 

F-16 

10 

12.0 

0.7 

F-111 

9 

10.8 

0.6 

Large  aircraft 

AC-130U 

6 

12 

0.5 

B-1 

3 

3.6 

0.4 

C-17 

1 

1.2 

0.3 

C-130 

1 

1.2 

0.3 

C-141 

1 

\2 

0.3 

E-3 

5 

6.0 

0.4 

EC-130 

20 

24.0 

1.6 

JSTARS 

3 

6.0 

0.4 

MC-130CT 

12 

14.4 

1.0 

CV-22 

12 

14.4 

1.0 

B-2 

2 

2.4 

0.3 
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The  values  in  Table  10  were  used  to  calculate  average  deployment  costs  per  week 
for  tactical  and  large  aircraft  of  $58,000  and  $79,000,  respectively.  (Details  of  this 
derivation  are  in  Appendix  D.) 

The  F-22  Systems  Program  Office  (SPO)  estimated  that  it  would  cost  $309  million 
more  in  constant  FY  1992  dollars  to  deploy  to  ACETEF  that  portion  of  the  F-22's 
EC/avionics  ground  test  program  involving  an  ISTF  than  to  conduct  that  ground  testing  at 
Edwards,  where  flight  testing  and  other  ground  testing  will  be  conducted.  This  estimate 
included  the  cost  to  construct  a  systems  integration  laboratory  (SIL)  with  two  avionics 
bench-test  lines  at  Patuxent  River  NAS  adjacent  to  ACETEF  and  its  chamber. 

The  AF  rationale  for  this  estimate  was  that  the  F-22  is  a  major  new  development 
that  incorporates  the  most  technologically  advanced  electronic  combat/avionics  systems 
ever  undertaken  by  the  DoD.  As  a  result,  the  Air  Force  considered  it  essential  that  a  SIL 
(or  hardware-in-the-loop)  facility  be  collocated  with  the  ISTF  if  the  ISTF  is  not  collocated 
with  the  flight  test  program.  The  $309  million  represents  the  total  cost  to  the  F-22  program 
associated  with  build-up  and  26  months  with  an  F-22  avionics  test  aircraft  in  an  anechoic 
chamber  at  ACETEF.  At  the  other  end  of  the  spectrum,  similar  deployment  to  ACETEF  of 
previous  operational  aircraft  for  tests  of  limited  duration  and  scope  without  a  collocated  SIL 
was  estimated  to  cost  only  $15  million. 

IDA  developed  a  "middle  ground"  deployment  estimate  of  $148  million,  which 
included  building  a  single-line  SIL  at  ACETEF  with  fewer  people,  fewer  spares,  less 
equipment,  and  less  duplicative  flight  testing  than  included  in  the  F-22  estimate.  We 
believed  this  estimate  to  be  a  reasonable  balance  between  risk  and  affordability.  Figure  10 
shows  the  impact  on  cost  of  using  the  various  deployment  estimates. 

4.  Total  Systems  Costs 

Table  1 1  pulls  together  the  investment,  operating,  and  deployment  costs  derived  in 
the  previous  tables  to  construct  the  20-year  systems  cost  of  each  of  the  four  testing 
alternatives.  The  20-year  costs  are  the  sum  of  investment  plus  20  years  of  additive 
operating  and  deployment  costs.  We  chose  a  20-year  cost  horizon  because  of  the  long  life 
of  the  resources  we  are  analyzing  (test  chambers  and  functions). 

As  already  discussed,  the  operating  and  deployment  costs  depend  partly  on  the 
number  of  .shifts  that  the  chambers  are  operated,  and  the  extent  of  the  Air  Force  testing 
program  that  is  deployed  to  ACETEF. 

Cost  estimates  for  the  alternatives  using  the  three  deployment  estimates  are 
summarized  in  Table  1 1.  The  medium  (IDA)  estimate  is  the  base  case,  with  the  high  (SPO) 
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T - 1 - 

WNh  Air  Force  WHh  IDA  With  MMmuin 

Deployment  Cost  Deployment  Cost  Deployment  Cost 

B  Currant  ACETEF  H  ACETEF  with  LAC 

O  Expanded  ECrr  Q  Both  LAC  and  ECIT 


Note:  Costs  include  investment  and  20-year  operations  costs. 

Figura  10.  Undiscountad  Costs  of  Altarnativss 


Tabia  11.  Costs  and  Effactivanass  of  Tasting  Altarnativss 


20- Year  Systems  Cost  (Millions  of  FY  1992  Dollars) 


Alternative 

Testing  Sboitfiill 
(Shift-Yeais) 

Total  Cost 

Discounted 

IgfillCTTl 

Thereafter 

Low 

-  Medium* 

High 

Low 

Medium* 

High 

1  Ejqianded  ACb'lbi' 

4.79 

4.47 

132 

262 

428 

75 

167 

281 

2  ACETEF -t- LAC 

3.29 

2.97 

324 

454 

620 

165 

249 

352 

3  ACETEF  +  ECIT 

2.29 

1.97 

486 

486 

486 

264 

264 

264 

4  AIL  2  + Alt.  3 

0.79 

0.47 

589 

589 

589 

328 

328 

328 

*Base  csK. 


B.  EFFECTIVENESS  ANALYSIS 

A  measure  of  effectiveness  of  each  testing  alternative  is  the  proportion  of  the  testing 
program  it  can  perform.  For  example,  if  the  Navy  and  Air  Force  together  have  a  testing 
requirement  of  7  shift-years  and  the  testing  alternative  can  deliver  four  shift-years,  there  is  a 
shortfall  of  at  least  3  shift-years.  The  smaller  the  shortfall,  the  more  effective  the 
alternative. 
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In  estimating  the  ability  of  only-ACETEF  alternatives  to  handle  both  Navy  and  Air 
Force  testing  programs,  we  made  the  simplifying  assumption  that  capacity  would  be 
divided  between  the  Services.  In  alternatives  that  had  ISTFs  at  both  Patuxent  and 
Edwards,  each  location  would  give  first  priority  to  the  testing  programs  of  its  own  Service. 
Thus,  the  ACETEF  facility  would  give  first  priority  to  the  Navy  test  program,  and  perform 
Air  Force  testing  with  whatever  excess  capacity  was  left  over.  Edwards  would  give  first 
priority  to  Air  Force  programs.  The  assumption  makes  no  judgment  about  the  relative 
value  of  the  respective  programs.  The  effectiveness  for  each  alternative  is  measured  by  the 
ability  of  the  alternative  to  handle  the  total  program.  Navy  and  Air  Force  combined. 

With  one  exception,  we  assumed  that  Air  Force  tactical  aircraft  require  a  small 
chamber  and  Air  Force  large  aircraft  require  a  large  chamber.  The  exception  is  the  F-22. 
Although  a  single  F-22  would  fit  in  a  small  chamber,  the  F-22  SPO  states  the  requirement 
to  group  several  aircraft  together  for  chamber  testing,  in  order  to  test  interactions  between 
aircraft  This  would  requite  a  large  chamber.  In  Alternative  1,  in  which  there  is  no  large 
chamber,  we  assumed  that  the  F-22  would  be  tested  as  a  single  aircraft  at  ACETEF  using 
the  existing  small  chamber. 

C.  SUMMARY  OF  COST  AND  EFFECTIVENESS 

In  Figure  1 1  we  attempt  to  show  costs  and  capacity  beyond  that  already  spent  and 
programmed  for  ACETEF.  The  scale  and  the  stacked  bar  to  the  left  of  each  series  shows 
the  cost  of  each  alternative.  The  scale  and  the  bar  to  the  right  of  each  series  shows 
additional  capacity  purchased.  Figure  12  shows  the  same  information  using  dollars 
discounted  at  10%  per  annum  over  the  cost  stream,  the  usual  measure  for  economic 
analyses  in  DoD. 

Tables  12  and  13  display  the  same  data  in  tabular  form.  The  more  we  spend,  the 
more  effectiveness  we  obtain,  as  measured  by  the  percentage  of  the  testing  program  that 
can  be  accomplished.  Note  that  in  each  successive  alternative  the  cost  per  unit  of  added 
capacity  is  lower.  Using  the  medium  (IDA)  F-22  deployment  costs  and  discounted  dollars 
(Table  13),  as  we  move  from  the  first  through  the  fourth  alternative,  the  20-year  discounted 
costs  about  double  (increase  from  $167  to  $328  million),  but  the  percentage  of  the 
Services'  testing  requirements  we  can  perform  (after  the  initial  5-year  period)  increases 
greatly,  from  36%  to  94%.  Similar  tables  showing  sensitivity  analyses  using  both  higher 
and  lower  F-22  deployment  costs,  and  higher  and  lower  Service  requirements  for  testing, 
are  in  Appendix  E. 
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Costs  (MiWons  ot  FY  1992  Dollars)  Costs  (Millions  of  FY  1992  Dollars) 


Bcpandad 

ADETEF 


ACETEF 
Wifi  LAC 


ACETEF 

WithECrr 


ACETff 
With  LAC 
and  ECrr 


n  Invastmant 
Q  20*Y»ar  Operations 


^  Daploymant  (IDA  Estimata) 
I  Additoral  EC  Tad  Naads  Mat 


Figura  11.  Costs  Versus  Needs  Met  (Constant  FY  1092  Dollars) 


Expmdad 

ACETEF 


ACETEF 
With  LAC 


ACETEF 

WKhECIT 


ACETEF 
With  LAC 
andECIT 


□  Invastmant 
O  20-Yaar  Oparatlons 


Daploymant  (IDA  Estimata) 
I  Addttonal  EC  Tast  Naads  Mat 


Figure  12.  Costs  Versus  Needs  Met  (Discounted  FY  1992  Dollars) 
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Additional  Capacity  (Peicant)  Additional  Capacity  (Parcant) 


TabI*  12.  Naads  and  Constrainad  Costa  with  IDA  Dapioymant  Estimata 
(Millions  of  Constant  FY  1992  Dollars) 


Alteniative43escriptioa 

Percenter' 
Needs  Met 

Increase  in 
Needs  Met 

Additive 

Costs* 

Cost  Per 
Percoitttf 
Added 
Needs  Met 

0 

Baseline  ACETEF 

22% 

0% 

$60 

NA 

1 

Provide  2.5  Shifts  at  ACEIEF 

36% 

14% 

$262 

$18.7 

2 

Build  LAC  at  Patuxent  with  4  Shift  Capacity 

57% 

35% 

$454 

$13.0 

3 

Build  EOT  at  Edwards  (and  ail  1  ACETEF) 

72% 

50% 

$486 

$9.7 

4 

Build  LAC  and  ECIT  (Alt  2  -t-  A1l3) 

94% 

72% 

$589 

$8.2 

a 

Uses  IDA  estimate  with  fiscally  constrained  F-22  deployment.  Costs  include  future  investmenu,  20-years  of 
operatkins  and  deployments.  Constant  FY  1992  dollars  are  undiscounted. 

TabI*  13.  Naada  and  Constrainad  Costs  with  IDA  Dapioymant  Estimata 
(Millions  of  Discountad  1992  Dollars) 

Alteroative/Descripdoa 

Percent  of 
Needs  Met 

Increase  in 
Needs  Met 

Additive 

Costs® 

Cost  Per 
Percoitof 
Added 
Needs  Met 

0 

Baseline  ACblliF' 

22% 

0% 

$50 

NA 

1 

Provide  2.5  Shifts  at  ACETEF 

36% 

14% 

$167 

$11.9 

2 

Build  LAC  at  Patuxent  with  4  Shift  Capacity 

57% 

35% 

$249 

$7.1 

3 

Build  EOT  at  Edwards  (and  AIL  1  ACETEF) 

72% 

50% 

$264 

$5.3 

4 

Build  LAC  and  EOT  (AIL  2  *  A1l3) 

94% 

72% 

$328 

$4.6 

*  Uses  IDA  estimate  with  fiscally  coostrained  F-22  deployment  Cosu  include  future  investments,  20>years  of 
operations  and  deployments.  FY  1992  doUan  are  disrounted  at  10%  per  annum. 


Note  that  the  second  and  third  options  are  relatively  close  together,  and  relatively 
separated  from  Alternatives  1  and  4,  in  terms  of  both  cost  and  effectiveness.  The  fact  that 
Alternatives  2  and  3  are  similar  in  cost  is  not  intuitive;  it  arises  from  the  fact  that  whereas 
building  a  new  chamber  at  Patuxent  costs  less  than  improving  the  BAF  at  Edwards  AFB, 
there  would  be  considerable  cost  in  deploying  a  new-development  Air  Force  aircraft  to 
Patuxent  for  flight  test. 

A  measure  of  merit  in  both  Table  12  and  Table  13  is  the  "cost  per  percent  of  added 
needs  met.  In  discounted  dollars  (Table  13),  we  see  the  cost  of  additional  needs  met 
decreasing  from  Alternatives  1  to  2  to  3  to  4  because  each  adds  capacity  at  a  greater  rate 
than  it  increases  cost  In  both  constant  and  discounted  dollars,  the  most  cost-effective 
alternative  under  this  measure  of  merit  is  Alternative  4.  Since  one  can  get  to  Alternative  4 
either  via  Alternative  2  or  Alternative  3,  it  would  appear  that  the  prudent  choice  might  be  to 
select  Alternative  3  (ECU')  as  the  first  step,  as  it  yields  a  substantially  greater  payoff  for 
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relatively  small  additional  cost  Thoe  are  other,  unquantifiable  considerations  (described  in 
Section  Vn)  that  also  affect  the  decision. 
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VIL  OTHER  CONSIDERATIONS 


This  section  details  a  number  of  considerations  that,  while  unquantiflable,  we  felt 
were  worthy  of  mention  here.  Most  support  tl»  acquisition  of  additional  installed  systems 
test  capability  at  Edwards  AFB,  although  some  reinforce  the  concentration  of  expertise  at 
Patuxent  NAS. 

A.  SERVICE-UNIQUE  REQUIREMENTS 

The  Naval  Air  Warfare  Center  (NAWC)  has  as  its  primary  mission  the  support  of 
the  Navy's  fleet  air  arm.  If  the  principal  job  of  the  ACETEF  activity  over  an  extended 
period  of  time  were  to  support  higher-priority  Air  Force  aircraft  programs,  would  fleet 
support  suffer?  For  example,  the  F-22  requirement  of  26  months  in  the  chamber  over  a 
four-  (M*  five-year  period  would  monopolize  that  facility  and  potentially  all  of  its  labs.  More 
importantly,  the  talents  of  the  best  ACETEF  personnel  would  be  solving  F-22  problems. 
Who  woiics  on  Navy  systems  and  problems  during  this  period?  On  the  other  hand,  if  the 
F-22  were  developed  using  ACETEF  exclusively,  who  supports  block  upgrades  and  needs 
that  arise  later  during  AF  operations?  The  expertise  and  experience  to  do  so  will  not  be 
readily  available  within  the  Air  Force.  Operational  systems  are  currently  the  only  ones  in 
test  at  ACETEF. 

B .  CONSIDERATIONS  AFFECTING  FUTURE  ISTF  WORKLOAD 

1 .  Impact  of  Declining  Defense  Budgets 

With  the  "drawdown"  in  defense  spending,  is  it  prudent  for  the  test  community  to 
undertake  investments  that  significantly  increase  its  capacity?  If  DoD  can  test  smarter  with 
new  facilities,  where  and  what  are  the  resulting  savings?  Can  we  reduce  test  flying  hours 
or  are  the  savings  only  in  risk  reduction?  These  policy  questions  are  outside  the  scope  of 
this  study  but  need  to  be  considered. 

2.  New  Defense  Acquisition  Strategy 

A  fact  sheet  on  Deputy  Secretary  of  Defense  Atwood's  new  approach  to  Defense 
acquisition  states  that  "particular  efforts  must  be  made  to  eliminate  cost  growth,  schedule 
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delays  or  trial-and-error  during  production.  The  Department  will  be  able  to  proceed  to 
production  of  a  weapon  system  only  after  thoroughly  demonstrating  the  effectiveness  of  the 
technologies  involved....  Accordingly  the  Department  will  emphasize  technology 
demonstration  and  prototype  evaluation  programs."  It  further  states  that  advanced 
technology  is  to  be  incorporated  in  new  or  existing  weapon  systems  only  if  "the  technology 
and  associated  systems  are  thoroughly  tested  and  proven"  [2].  Such  a  policy  clearly 
requires  a  strong  electronic  combat/avionics  test  capability. 

3.  Increased  Demand  for  Ground  Testing 

Experts  within  both  the  program  management  and  test  communities  believe  that 
future  demand  for  the  ground  testing  of  electronic  combat/avionics  will  increase  due  to  the 
increasing  complexity  of  avionics  and  the  need  for  a  significant  amount  of  ground  testing 
from  early  development  through  operations.  Figures  13  and  14  show  the  growth  of 
avionics  (1)  as  a  percentage  of  aircraft  flyaway  cost  and  (2)  in  terms  of  lines  of  software 
code  in  air  and  space  systems.  As  electronic  combat  and  other  avionics  packages  become 
more  integrated  and  complex,  either  in  new  aircraft  or  in  retrofits,  better  ground  facilities 
are  needed.  For  example,  the  multi-spectral  threat  cannot  be  tested  on  open  ranges. 


Source:  Longbraks,  Ronald  B.  'Avionica  Acquisition,  Trends  and  Future  Appioacttes.' 

NATO  Advisory  Group  for  Aerospace  Research  and  Development  (AGARD), 
Conference  Proceedings  No.  424:  Flight  Vehicle  Development  Time  and  Cost 
Reduction,  pp.  11-1  to  11-11. 

Figure  13.  Avionics  Growth  as  a  Percentage  of  Aircraft  Flyaway  Cost 
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Figur*  14.  Avionics  Growth  in  Terms  cf  Lines  of  Software  Code 


4.  Lead  Times  Necessary  for  Facilities  to  Test 
New  Advanced  Technologies 

Figure  IS  shows  typical  lead  times  in  acquiring  facilities  through  the  military 
construction  process.  Planning  and  administrative  times  average  about  3.5  years  with  1.5 
years  more  in  actual  construction  to  acquire  the  required  test  capability.  Hiis  schedule 
dictates  that  facility  constructitm  needs  to  be  started  long  before  the  facility  is  needed. 

5.  Availability  of  Experienced  Work  Force 

Because  the  ACETEF  work  force  is  already  experienced,  building  an  expanded 
capability  at  Patuxent  River  would  be  easier  and  quicker  than  starting  an  essentially 
capability  at  Edwards. 

6.  Potential  to  Meet  Workload  Surges 

Even  if  one  ISTF  had  the  capacity  to  satisfy  critical  DoD  requirements,  there  would 
be  value  in  having  two  to  meet  wortdoad  surges  or  the  catastrophic  loss  of  one.  This  is  a 
significant  concern  considering  the  cost  to  fix  problems  later  due  to  the  lack  of  a  T&E 
capability  or  the  cost  to  delay  the  schedule  of  a  major  program. 
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Source:  Derived  from  historical  data  by  Arnold  Engineering  Development  Center,  1988. 


Figure  15.  Construction  Lead  Times 

C.  PROJECT  RELIANCE:  THE  NEED  TO  MAKE  IT  WORK 

"Project  Reliance”  is  an  initiative  of  the  Joint  Commanders  Group,  Test  and 
Evaluation,  to  promote  efficient  muld-Service  use  of  T&E  investment  and  facilities  and  to 
eliminate  unwarranted  duplication  of  T&E  facilities.  Over  the  course  of  a  year,  the 
Electronic  Warfare  (EW)  T&E  Reliance  Study  Group  assessed  the  spectrum  of  test  support 
resources:  digital  models  and  computer  simulation,  integration  laboratories,  hardware-in- 
the-loop  test  facilities,  open  air  ranges,  and  measurement  facilities  as  well  as  the  installed 
systems  test  facilities  (including  anechoic  chambers)  that  are  the  focus  of  this  report  The 
Study  Group  recommended  in  part  [3]: 

•  The  Navy  should  be  designated  as  lead  Service  for  ISTFs. 

•  An  ISTF  EW  T&E  Resource  Master  Plan  should  be  developed  by  the  Navy 
assisted  by  the  Army  and  the  Air  Force. 

•  The  ISTF  EW  T&E  Resource  Master  Plan  should  consider  (1)  developing 
ACETEF  into  the  highest  capability  (Category  I)  ISTF,  (2)  authorizing 
planning  for  a  larger  anechoic  facility  at  NAWC-AD,  but  deferring  construction 
until  "future  workload  requirements  are  demonstrated  and  the  DoD  budget 
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picture  stabilizes,"  (3)  developing  a  large,  enhanced-capability  (Category  11) 
ISTF  at  Edwards  with  emphasis  on  support  for  new-generation  systems. 

These  reconunoidations  endorse  the  need  to  upgrade  the  electronic  combat/avionics 
ground  test  capabilities  both  at  Patuxent  and  at  Edwards. 

With  the  Project  Reliance  Memorandum  of  Understanding  signed,  the  framework  is 
in  place  to  begin  a  coordinated  multi-Service  approach  to  proceed  with  improvements  while 
minimizing  potential  duplication.  It  is  our  opinion  that  the  Reliance  process  should  be 
given  the  opportunity  to  work. 

D.  TEST  FACILITY  DUPLICATION  VERSUS  OPTIMIZATION  FOR 
WEAPON  SYSTEM  ACQUISITION 

1 .  Relative  Costs  of  Test  Facilities  and  Weapon  Systems 

In  order  to  produce  the  strongest  defense  for  a  given  level  of  resources,  the  entire 
acquisition  process  must  be  considered.  This  may  require  some  duplication  of  test  assets  in 
order  to  minimize  the  cost  of  weapon  system  acquisition.  A  theoretical  example  is 
procuring  two  $10  million  test  instrumentation  packages  in  order  to  prevent  production 
delays  costing  $100  million. 

The  cost  of  the  new  capabilities  we  are  talking  about  in  this  paper,  while 
substantial,  are  not  large  compared  to  the  cost  of  the  systems  they  are  designed  to  support 
For  example,  the  FY  1993  unit  cost  of  the  B-2A  is  $1  billion,  the  C-17,  $335  million;  even 
"end  of  the  production  line"  aircraft  such  as  the  F-16  and  C-130  have  FY  1993  unit  costs 
over  $30  million  [56].  A  relatively  small  test  investment  may  leverage  huge  acquisition 
investments  and  significantly  reduce  overall  acquisition  costs. 

2 .  Potential  for  Recovery  of  Test  Facility  Investment  in  Peace  and  in  War 

An  excellent  opportunity  exists  to  recover  test  capability  investments  during 
peacetime  through  avoidance  of  test  and  schedule  delays  during  acquisition,  reduced 
requirements  for  flight  testing,  and  prevention  of  aircraft  losses  during  test  and  operations. 
The  cost  of  new  test  capabilities  might  also  be  recovered  by  reducing  weapon  systems 
acquisition  costs  (e.g.,  less  test  flight  hours,  reduced  procurement  quantities).  Another 
example  is  the  potential  to  avoid  a  many  of  the  software  fixes  that  often  occur  during 
production  and  q)eration. 
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Figure  16  shows  results  of  a  study  in  which  the  median  relative  cost  to  fix  an  error 
during  development  test  was  found  to  be  almost  an  order  of  magnitude  lower  than  it  was 
during  operation.  Thus  very  large  savings  accrue  from  discovering  software  errors  early  in 
an  acquisition  program.  Ground  testing  in  facilities  such  as  an  ISTF  are  the  only  means  to 
test  many  of  the  parameters  in  the  software  for  installed  systems.  Inasmuch  as  it  is 
expected  that  software  could  be  a  significant  cost  for  the  F-22  or  any  new  aircraft,  the 
opportunities  for  cost  avoidance  through  ISTF  are  substantial. 


Phase  in  which  Error  Detected 

Source:  Boohm,  Bany  W.  ‘Softwaie  Rnk  ManegMMnL*  TRW,  Inc.,  Wuhinglon,  D.C.,  1969,  p.  3. 


Figure  16.  Escalation  of  Software  Rework  Costs  with  Phase 
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Being  able  to  test  war  reserve  modes  thoroughly  in  a  secure  environment  and 
improve  crew  responses  in  war  scenarios  likewise  has  potential  for  big  returns  on 
investment  should  war  occur. 

E.  UNANSWERED  TECHNICAL  QUESTIONS^^^  ^ 

During  the  cour^^pftJuTmvesdgati^  several  technical  questions  arose  to  which 
no  answers  were  found.  Such  questions  should  be  addressed  when  developing  a  program 
for  a  future  large  chamber  construction  at  Patuxent  River  NAS  en^when  developing  the 
upgrartes  to  the  existing  large  chamber  at  Edwards  AFB.  The  questions  are: 

•  How  does  one  test  multi-spectral  systems?  An  answer  is  required  before 
building  a  Category  I  installed  systems  test  facility. 

•  Far-fleld  signal  generation  versus  signal-injection:  How  far  is  far  enough? 

How  many  free-flow  signal  generators  are  needed  to  simulate  reality?  It  is 
expected  that  injection  techniques  will  also  be  needed  and  the  AF  should  plan 
for  that  (Further  discussion  of  far-field  testing  is  in  Appendix  G.) 

■  Can  existing  threat  simulation  models/capabilities  around  the  country  be  moved 
to  Edwards  to  upgrade  the  existing  chamber?  Savings  accrue  by  using 
"standard"  ciq)abilities  already  proved. 

At  the  request  of  DT&E,  we  also  investigated  whether  the  Grumman-operated 
anechoic  chamber  at  Calverton,  NY,  could  be  used  in  lieu  of  additional  facilities  at  Edwards 
or  Patuxent  River.  The  short  answer  is,  "no."  A  discussion  is  indued  in  Appendix  G. 


VIII.  FINDINGS 


This  section  presents  a  summary  of  the  findings  of  our  analyses. 

A.  INCREASING  INSTALLED  SYSTEMS  TEST  FACILITY 
CAPABILITY— PROS  AND  CONS 

1 .  Current  capability  meets  only  22%  of  stated  Service  electronic  combat/avionics 
ISTF  requirements. 

2.  Ground  testing  is  the  only  way  to  do  much  EC  testing.  Reasons  for  this 
include: 

a.  Security  considerations. 

b.  The  number  of  test  parameters. 

c.  Ability  to  test  earlier  in  the  program  and  mote  safely. 

3.  Specific  advantages  of  ISTFs  over  less  capable  facilities  include  five  types  of 
testing: 

a.  Tests  that  require  the  test  article  to  be  installed  in  an  aircraft  in  order  to  be 
evaluated. 

b.  Tests  that  requite  levels  of  stimulation  greater  than  those  available  in  lesser 
facilities. 

c.  Tests  that  require  radiating  elements  to  operate  in  an  "in-flight" 
configuration. 

d .  Tempest  tests  of  specific  installed  hardware. 

e.  Tests  that  requite  absolute  security  of  RF  signals,  either  threat  or  friendly. 
Examples  of  the  types  of  testing  uniquely  appropriate  to  ISTFs  are  in 
Appendix  G. 

4.  ISTF  testing  is  cost-effective. 

a.  Test  facility  co'ts  are  small  relative  to  total  systems  acquisition  costs. 
Testing  makes  acquisition  decisions  smarter  and  prevents  delays  and  costly 
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retrofits  that  may  be  several  orders  of  magnitude  more  costly  to  perform 
once  production  and  fielding  has  taken  place. 

b.  Both  AF  studies  and  IDA  studies  based  on  Navy  data  indicate  ISTF 
development  and  operation  may  cost  only  one-eighth  the  cost  of  performing 
the  same  tests  by  flying.  Three-fourths  of  software  problems  are  said  to  be 
resolvable  on  the  ground.  Use  of  ISTFs  minimizes  the  expensive  fly-fix-fly 
approach  to  testing. 

c.  Ground  testing  is  more  efficient  It  allows  control  of  experiment  and  rapid 
repetition  of  test  conditions.  It  reduces  test  costs  and  accelerates  test 
schedules. 

d.  Ground  testing  allows  more  effective  use  of  flight  test  time  by  focusing  on 
risk  areas.  It  makes  flight  testing  and  later  operations  safer,  which  may 
save  the  aircraft,  crew,  and  mission,  if  problems  are  solved  on  the  ground 
first . 

5 .  These  counter-arguments  to  more  ISTF  capability  were  also  found: 

a.  Current  workload  data  do  not  argue  well  for  increases  to  capacity. 
ACETEF  currently  operates  at  1.5  shifts  but  could  go  to  2.5.  The  BAF 
operates  at  less  than  1  shift 

b.  The  DoD  budget  is  declining.  Future  defense  needs  are  uncertain. 

B .  USING  AN  IMPROVED  ACETEF  TO  SATISFY  AIR  FORCE 
REQUIREMENTS— PROS  AND  CONS 

1 .  Advantages  of  improving  ACETEF  include: 

a.  ACETEF  is  without  question  the  best  DoD  ISTF  for  performing 
EC/avionics  ground  testing  at  the  present  time. 

b.  ACETEF  has  unused  capacity  now  and  can  expand  its  operations  with  the 
least  additional  investment  and  operations  costs. 

c.  ACETEF  has  the  most  competent  and  experienced  EC/avionics  ISTF  test 
personnel— over  10  years  of  hands-on  development  of  capability  and  test 
performance.  In  fact,  ACETEF  has  the  only  proven  capability  in  this  area. 
The  expertise  to  best  support  the  F-22  resides  at  ACETEF. 
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2.  Negatives  regarding  the  choice  of  ACETEF  for  expansion  and  AF  testing 

include: 

a.  The  AF  philosophy  of  testing  is  predicated  on  the  close  proximity  of  ground 
testing  and  flight  testing  (the  Combined  Test  Force  concept). 

b.  The  Chesapeake  Test  Range  at  Patuxent  NAS  is  considered  to  be  too  small 
and  congested  for  additional  high-performance  AF  flight  testing. 

c.  AF  requirements  drive  the  need  for  additional  ISTFs. 

d.  The  cost  of  deploying  the  F-22  to  ACETEF  for  electronic  combat/avionics 
ground  testing  offsets  the  cost  of  developing  an  expanded  ECIT  at  Edwards 
AFB. 

e.  Building  ECIT  would  add  more  EC  test  capacity  than  expanding  ACETEF, 
and  at  less  cost  per  unit  of  added  capacity. 

C.  DEVELOPING  AN  ISTF  CAPABILITY  AT  EDWARDS  AFB— PROS 
AND  CONS 

1 .  Advantages  of  improving  ECIT  include: 

a.  Edwards  currently  has  the  world’s  largest  anechoic  chamber,  the  BAF. 
This  is  the  only  current  chamber  with  potential  for  far-field  signal  testing, 
testing  multiple  small  aircraft  and  testing  large  aircraft. 

b.  Potential  for  any  significant  EC/avionics  test  capability  using  the  BAF 
cannot  be  met  without  significant  investment  in  supporting  infrastructure. 

c.  The  ECIT  would  be  close  to  large  and  relatively  uncongested  ranges  for  EC 
and  other  flight  testing.  This  fits  the  AF  concept  of  single-site  testing. 

d.  The  Air  Force  has  a  wide  variety  of  aircraft  and  missions  that  can  benefit 
from  testing  in  ISTF.  All  of  these  aircraft,  many  of  which  are  larger  than 
any  in  the  Navy,  are  flight  tested  at  Edwards.  Development  of  an  ECIT 
capability  over  time  at  the  primary  Air  Force  flight  test  center  makes  sense. 

e.  In  conjunction  with  the  small  chamber  already  at  ACETEF,  building  ECIT 
to  support  the  BAF  is  more  cost-effective  than  building  a  large  chamber  at 
Patuxent. 
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f.  Building  at  Edwards  would  provide  a  second  government  source  and 
competition  for  ISTF  capability,  and  the  stated  requirements  indicate  the 
additional  capacity  is  justified. 

2 .  Negatives  regarding  ECFT  include: 

a.  The  ECIT  program  plan  is  incomplete.  We  are  not  sure  what  specific 
funding  levels  might  buy. 

b.  Most  of  the  work  force  for  an  ECIT  has  yet  to  be  recruited  and  gain 
experience. 

c.  It  will  be  difficult  for  ECIT  to  be  fully  ready  in  time  to  support  early  testing 
in  the  F-22  schedule. 

D.  SUMMARY 

The  IDA  analysis  shows  that  building  HTT  rather  than  a  large  chamber  at  ACETEF 
is  the  most  cost-effective,  lowest  risk  alternative.  While  both  options  would  add  the 
capability  to  test  large  aircraft,  the  ECIT  at  Edwards  would  provide  the  most  flexibility  to 
adjust  to  an  uncertain  workload  and  would  add  more  total  capacity  at  less  cost  per  unit  It 
supports  the  current  AF  philosophy  of  collocating  ground  and  flight  testing,  and  provides  a 
needed  additional  source  of  test  capability.  Further,  it  leverages  the  investments  already 
made  in  the  BAF  and  provides  an  opportunity  to  pursue  the  concept  of  far-field  signal 
testing. 
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GENERIC  ECIT  COSTS  AND  FUNDING 

The  Air  Force  provided  a  detailed  cost  estimate  for  developing  the  generic  ECIT 
capability  that  was  intended  to  satisfy  the  basic  requirements  of  many  different  programs 
and  serve  as  the  foundation  for  building  additional  program-unique  requirements.  Such  a 
generic  type  of  capability  is  typically  budgeted  and  funded  using  institutional  funds  of  the 
responsible  T&E  organization  or,  in  this  particular  case,  the  AF  Flight  Test  Center  at 
Edwards  AFB. 

During  this  study,  we  received  documentation  from  and  participated  in  discussions 
with  various  AF  personnel  about  the  technical  content  and  cost  of  the  generic  ECIT 
capability.  The  generic  capability  was  usually  referred  to  as  the  "$60  million  program." 
However,  the  briefings  and  supporting  documentation  we  received  did  not  adequately 
show  a  clear  track  between  the  various  cost  and  funding  profiles.  Table  A-1  is  a  summary 
of  the  different  profiles. 

Table  A-1.  Generic  ECIT  Program  and  Funding  as  of  1992  Budget  Submission 

(Millions  of  FY  1992  Dollars) 


FY92 

FY93 

FY94 

FY95 

FY96 

FY97 

Total 

Program  Requirements 

Investment 

7.8 

14.2 

12.9 

7.5 

7.7 

3.2 

53.3 

Program  Expenses 

3.1 

3.1 

3.1 

3.1 

3.1 

3.1 

18.6 

Total  Generic  ECIT  Program 

Funding 

10.9 

17.3 

16.0 

10.6 

10.8 

6.3 

71.9 

Investment  (PE  64735) 

6.9 

6.9 

6.9 

6.9 

6.9 

6.9 

41.4 

Program  Management  (PE  64735) 

3.1 

3.1 

3.1 

3.1 

3.1 

3.1 

18.6 

Subtotal  (PE  64735) 

10.0 

10.0 

10.0 

10.0 

10.0 

10.0 

60.0 

Investment  (PE  64755) 

3.4 

4.9 

8.3 

Subtotal  (res  64735  &  64755) 

13.4 

14.9 

10.0 

10.0 

10.0 

10.0 

68.3 

Unfunded  Requirements  (Investment) 

-2.4 

2.4 

6.0 

0.7 

0.8 

-3.7 

3.8 

Total  Generic  ECTT  Funding 

11.0 

17.3 

16.0 

10.7 

10.8 

6.3 

72.1 

Note:  Slight  differences  between  program  requirements  and  funding  are  due  to  rounding. 

In  the  program  requirements  section  of  the  table,  we  show  the  AF  estimate  of  the 
current  ECIT  program  which  totals  about  $72  million  over  a  6-year  period.  The  program 
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consists  of  $53  million  in  investment  assets  (e.g„  hardware  and  software)  and  $19  million 
in  costs  for  managing  the  development  and  implementation  of  the  ECIT  program. 

The  funding  section  shows  the  approved  program  as  contained  in  the  FY  1992 
President’s  Budget  prior  to  the  congressional  cut  that  bought  FY  1992  funding  to  zero. 
The  "$60  million  program"  consists  only  of  those  costs  budgeted  in  program  element  (PE) 
64735,  Range  Improvements,  to  include  investments  of  $41.4  million  and  program 
expenses  of  $18.6  million.  There  is  an  additional  funding  line  representing  investments  for 
PE  64755,  Development  Test  and  Evaluation,  of  $8.3  million.  This  leaves  a  funding 
shortfall  in  investments  of  $3.8  million.  The  combined  total  of  investments,  program 
expenses,  and  unfunded  investment  requirements  approximately  equals  the  estimated  AF 
costs  of  $72  million. 

The  IDA  estimate  of  the  cost  of  an  ECIT  capability  equivalent  to  that  programmed 
for  ACETEF  is  about  $168  million  in  constant  FY  1992  dollars  (see  Table  9,  Section  VI). 
The  shortfall  of  $100  million  would  be  borne  by  programs  requiring  speciEc  capabilities 
beyond  the  generic  ones,  notably  the  F-22.  Coincidentally,  this  estimate  is  relatively 
consistent  with  amounts  briefed  by  Edwards  AFB  on  5*6  February  1992.  Officials  at 
Edwards  stated  that  some  time  earlier  General  Yates,  Commander  of  AF  Systems 
Command,  had  directed  Edwards  to  program  $60  million  in  internal  Improvement  and 
Modernization  (I&M)  funds  for  the  generic  capability,  and  that  $90  million  would  be 
forthcoming  from  the  system  program  offices  for  program-specific  capabilities. 
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CONVERSION  TO  FY  1992  CONSTANT  DOLLARS 

All  then-year  costs  were  converted  to  FY  1992  dollars  for  analytical  purposes. 
Generally,  this  process  is  simple  and  straightforward.  Annual  costs  in  then-year  dollars  are 
converted  to  annual  constant  dollars  by  applying  the  appropriate  DoD  deflator  for  the 
specific  appropriation  and  fiscal  year.  Unfortunately,  the  relevant  costs  for  this  study  could 
not  always  be  broken  out  by  appropriation  and/or  fiscal  year.  As  a  result,  we  had  to 
develop  alternative  methods  for  estimating  costs  in  FY  1992  dollars.  We  used  the  standard 
DoD  deflators  (with  FY  1992  as  the  base  year)  for  total  obligation  authority  (TO A)  to 
compute  1992  base-year  dollars.  If  the  category  description  indicated  that  the  cost  was 
Military  Construction  (MILCON).  we  used  the  Navy  or  AF  construction  deflator.  All  other 
costs  were  converted  using  the  Navy  or  AF  RDT&E  deflator. 

NAVY  COSTS 

Table  B-1  displays  the  original  ACETEF  costs  as  provided  by  the  Navy,  the  data 
used  to  convert  those  costs  to  FY  1992  dollars,  and  the  estimates  expressed  in  FY  1992 
dollars.  The  Navy  provided  ACETEF  costs  for  existing,  programmed  (planned  and 
approved),  and  planned  but  not  approved  functions  and  related  facilities  (column  1). 
Section  IV  describes  these  functions. 

The  cumulative  cost  totals  for  FY  1992,  $249.45  million  (column  2),  and  FY  1995, 
$397.35  million  (column  4),  were  a  mixture  of  then-year  and  constant  dollars.  For  existing 
functions,  some  costs  incurred  in  FY  1989  and  prior  were  not  segregated  by  function  and 
fiscal  year.  For  these  costs  ($88.4  million),  we  assumed  they  were  incurred  beginning 
with  the  opening  of  the  anechoic  chamber  in  1983  and  continuing  in  seven  equal 
installments  through  1989.  Using  the  Navy-provided  total  then-year  costs  by  function 
through  1992  (and  1995),  we  calculated  the  relative  percentage  for  each  of  these  functions, 
multiplied  that  percentage  times  the  total  1989  costs,  and  divided  that  product  by  the 
average  RDT&E  deflator  for  1983-1989  to  produce  $107.66  million  in  FY  1992  doUar 
costs  for  that  period  (column  7). 
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Beginning  in  1990,  the  Navy  furnished  individual  fiscal  year  costs  by  source  of 
funding  (e.g.,  direct  project  support,  I&M,  CTEIP,  etc.)  but  did  not  break  them  out  by 
function  (although  facilities  costs  were  subdivided).  We  took  the  annual  funding 
(FY  1990,  $15.75  million;  FY  1991,  $12.8  million;  FY  1992,  $15.6  million),  multiplied 
each  by  the  relative  percentage  for  individual  functions  through  FY  1992  and  divided  by  the 
yearly  RDT&E  deflators  for  FY  1990  through  FY  1992.  We  then  added  the  FY  1989 
cumulative  amounts  (column  7)  and  the  individual  FY  1990-92  amounts  (columns  8,  9, 
and  10)  to  obtain  $  153.61  million  in  total  costs  through  FY  1992  in  FY  1992  dollars 
(column  11). 

We  computed  the  change  in  costs,  $65.20  million,  between  functions  for  the  1992 
and  1995  data  submitted  by  the  Navy  (column  5).  We  then  calculated  the  relative 
percentage  change  for  each  of  the  functions  (column  6),  multiplied  that  by  the  related 
funding  for  each  fiscal  year  (FY  1993,  $8.3  million;  FY  1994,  $20.6  million;  FY  1995, 
$26.5  million),  and  divided  by  the  appropriate  annual  RDT&E  deflator  to  arrive  at  FY  1992 
dollars  (columns  12, 13,  and  14).  Like  the  Navy,  we  calculated  total  costs  by  function  and 
facilities  through  FY  1992  and  through  FY  1995  of  $271.46  million  (column  11)  and 
$413.71  million  (column  16),  respectively. 

The  Navy  specified  that  the  small  anechoic  chamber  (SAC)  and  shielded  hangar 
(SH)  costs  represented  current  replacement  values.  We  accepted  those  amounts  as 
equivalent  FY  1992  dollars.  The  Navy  also  provided  an  estimate  of  $68  million  for  the 
proposed  large  anechoic  chamber  (LAC).  We  assumed  this  estimate  to  be  in  FY  1992 
dollars  because  it  agreed  with  our  calculation  using  the  last  detailed  Navy  estimate  of  $58 
million  in  FY  1988  dollars  and  adjusting  for  the  Navy  MILCON  deflator.  We  also 
arbitrarily  displayed  the  LAC  amount  in  FY  1994  because  the  FY  1992  construction 
program  is  currently  being  executed  and  the  President's  fT  1993  budget  submission  has 
already  been  determined. 

AIR  FORCE  COSTS 

The  AF  estimates  for  the  various  ECIT  functions  and  test  deployment  costs  were  in 
FY  1992  dollars.  No  conversion  was  required  for  these  costs.  However,  the  F-22 
program  office  estimate  of  the  deployment  costs  involved  in  testing  at  ACETEF  rather  than 
at  ECIT  was  submitted  in  then-year  dollars,  and  these  costs  were  was  converted  to  constant 
FY  1992  dollars  using  RDT&E  deflators. 
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Operating  costs  represent  the  resources  consumed  in  performing  the  routine  and 
recurring  tasks  associated  with  ISTF  function.  The  principal  cost  elements  are  labor, 
materiel  (supplies  and  equipment)  and  facility  support  (e.g.,  utilities,  security  service,  etc.). 
The  dominant  cost  element  is  labor  (both  government  and  contractor  personnel)  that  is 
required  to  develop  the  test  capability  and  to  perform  the  actual  test  function.  The  following 
subsections  describe  current  and  estimated  future  operating  costs  for  the  Navy's  ACETEF 
capability  and  the  AFs  ECIT  capability. 

ACETEF  OPERATING  COSTS 

The  Navy  did  not  provide  us  with  a  total  cost  estimate  for  operating  the  ACETEF 
facility  on  an  annual  basis.  The  Navy  stated  that  available  total  costs  included  other  support 
not  related  directly  to  anechoic  chamber  testing  (e.g.,  work  related  to  Strike  aircraft)  and 
that  the  costs  we  wanted  could  not  be  broken  out  and  shown  separately.  However,  the 
Navy  was  able  to  provide  operating  cost  data  for  two  laboratories  and  the  chamber  itself, 
manpower  totals  by  ACETEF  department,  and  marginal  manpower  estimates  for  increasing 
the  number  of  shifts  and  expanding  facilities. 

Specifically,  the  Navy  estimated  the  following  FY  1993  operating  costs  in 
thousands  of  then-year  dollars:  Electronic  Warfare  Integrated  Systems  Test  Laboratory 
(EWISTL)  $580.3;  manned  flight  simulator  (MFS),  $1,065.5;  and  the  anechoic  chamber, 
$420.  We  attempted  to  relate  these  costs  with  the  number  of  government  people  assigned 
and  to  the  asset  values  shown  for  these  activities,  in  order  to  develop  cost  estimates  for 
other  capabilities.  However,  we  were  unable  to  identify  any  consistent  relationships 
among  the  data  and  could  not  develop  any  other  cost-estimating  relationships. 

The  Navy  cunently  operates  an  average  of  about  1.5  shifts  per  day  annually  at  the 
ACETEF  facility.  The  Navy  estimated  that  no  additional  costs  other  than  to  programs 
would  be  incurred  to  support  a  2.5  shift  operation  in  existing  facilities.  The  additional 
throughput  would  be  achieved  through  revised  scheduling  of  people  and  equipment  If  the 
large  anechoic  chamber  (LAC)  were  constructed,  the  Navy  believed  that  an  additional  12 
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government  people  would  be  required  to  operate  the  LAC  and  about  another  10  test  or 
laboratory  personnel  would  be  needed.  Using  the  Navy  estimate  of  $120  thousand  for  a 
fully  loaded  contractor  work  year,  we  estimated  the  marginal  operating  costs  to  be  $2.64 
million.  These  costs  for  22  people  would  support  two  shifts  each  in  the  small  and  large 
chambers  and  the  related  four  laboratory  shifts.  We  estimated  the  cost  of  20  full  years  of 
operations  as  20  x  $2.64  =  $52.8  million  plus  an  initial  half  year  to  bring  up  the  operation 
($1.3  million),  $54.1  million  in  all. 

ECIT  OPERATING  COSTS 

The  AF  reported  that  the  current  annual  costs  to  operate  the  Benefield  Anechoic 
Facility  (BAF)  is  approximately  $5.5  million.  This  paid  for  1  shift  (8  hours  daily,  5  days 
per  week,  48  weeks  per  year)  consisting  of  5.25  hours  test  time  and  2.75  hours  for  routine 
maintenance.  The  AF  points  out  that  this  shift  structure  is  not  the  preferred  alternative  but 
one  that  is  largely  driven  by  funding  constraints.  Table  C-1  shows  the  major  cost  elements 
in  the  current  BAF  operating  program.  The  largest  element  is  labor,  which  represents 
almost  87%  of  the  total  program.  Labor  includes  civilian  pay.  Engineering  Technical 
Support  Services  contract,  business  management  office  overhead  included  in  the  contract, 
and  award  fee. 


Tabl«  C-1.  BAF  Annual  Oparating  Costs 
(Millions  of  FY  1992  Doilars) 


Equipment  (Over  15K} 

35 

0.6% 

Civilian  Pay 

1,139 

20.7% 

Travel 

25 

0.5% 

Equipment  Maintenance 

3 

0.1% 

Computer  Maintenance 

5 

0.1% 

Engineering  Technical  Support  Services  Contraa 

3,069 

55.8% 

Business  Management  Office  Overhead  (Clcrntract) 

295 

5.4% 

Award  Fee 

275 

5.0% 

Travel  Adminisuation 

10 

0.2% 

Supplies 

13 

0.2% 

Ginuxl  Service 

352 

6.4% 

Administrative  Supplies  &  Equipment 

79 

1.4% 

Utilities 

166 

3.0% 

Janitorial  and  Copying  Services 

34 

0.6% 

Total 

5,500 

100.0% 

The  AF  notes  that  the  more  desirable  shift  approach  would  be  a  full  8  hours  of 
testing  followed  by  8  hours  of  maintenance.  Major  periodic  maintenance  and 
facility/equipment  upgrades  were  not  inclutfed  in  the  $5.5  million  program.  An  increase  to 


2  shifts  for  testing  alone  would  consist  of  16  hours  of  testing  and  an  additional  8  hours  of 
routine  maintenance.  This  would  cost  about  $8  million.  This  scenario  also  delays  periodic 
maintenance  and  upgrades. 

The  AF  estimates  that  an  additional  $1.5  million  to  $1.9  million  would  be  necessary 
to  incorporate  full  periodic  maintenance  and  upgrades  in  the  desired  shift  configurations. 
SpeciHcally,  a  total  of  $7  million  would  be  required  to  run  1  full  shift  of  testing  and  1  full 
shift  of  full  maintenance  (routine  and  periodic  and  upgrades).  A  total  of  $9.9  million 
would  be  needed  to  perform  2  testing  shifts  and  1  full  maintenance  shift 

The  AF  indicated  additional  resources  would  depend  on  the  specific  aircraft  system 
being  tested  and  its  particular  testing  requirements.  Given  that  the  AF  intends  to  operate  the 
B  AF  with  or  without  ECIT,  we  did  not  consider  the  $5.5  million  program  to  be  a  marginal 
cost.  However,  we  included  the  additional  $4.4  million  needed  to  perform  2  test  shifts  and 
1  maintenance  shift  ($9.9  million  program)  as  an  additional  cost  to  operate  the  BAF  in  an 
ECIT  configuration.  We  viewed  this  3-shift  combination  to  be  the  equivalent  of  the 
2.5'Shift  ACETEF  operation. 

The  $4.4  million  in  costs  represent  the  additional  costs  to  perform  the  actual  testing. 
Additional  personnel  also  are  required  to  develop  the  test  capability.  In  the  absence  of  a 
specific  AF  estimate,  we  used  the  estimated  number  of  government  "developers"  currently 
at  ACETEF  (45  personnel)  that  can  support  up  to  2.5  shifts.  We  priced  these  45  people  at 
the  fully  loaded  contractor  rate  of  $130  thousand  per  man-year  for  a  total  of  $6.85  million 
in  additional  costs.  This  included  the  annual  program  expenses  of  $3. 1  million  associated 
with  the  ECIT  generic  program,  which  contained  "developer  costs"  during  three  years  of 
the  opi  ting  period  (FY  1996-98).  Total  marginal  operating  costs  were  estimated  at 
$1 1.25  million  ($6.85  million  +  $3.75  million)  to  support  the  ECIT  capability  operating  at 
2.5  shifts.  We  estimated  the  total  cost  of  20  years  of  full-up  operations  to  be  20  x  $1 1.25 
=  $225  million,  plus  half  a  year  of  the  $8.15  million  to  start  operations,  equals  $229.1 
million. 
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DEPLOYMENT  COSTS 

Experts  generally  agree  that  installed  systems  testing  and  flight  testing  are  most 
efficiently  and  effectively  accomplished  at  the  same  location.  This  permits  the  shared  use 
of  many  of  the  same  resources  needed  for  both  types  of  testing.  When  such  testing  is 
geographically  separated,  additional  costs  are  usually  incurred  for  moving  test  personnel 
and  materiel  (e.g.,  test  articles,  equipment,  spares,  etc.)  to  the  new  test  site. 

This  section  describes  the  methodology  and  estimated  incremental  costs  that  the  Air 
Force  ordinarily  would  incur  if  they  were  required  to  use  the  Navy's  ACETEF  facility 
rather  than  the  ECIT  facility  at  Edwards  AFB  where  development  flight  testing  will  be 
performed.  The  methodology  does  not  include  any  costs  estimates  for  any  additional 
materiel  that  may  have  to  be  (Mocured,  additional  labor  above  what  is  already  required,  and 
possible  schedule  delays.  The  potential  for  these  additional  resource  requirements  varies 
by  program  and  is  included  only  in  the  F-22  estimates. 

AF  ESTIMATES 

In  its  14  January  1992  submission  to  OSD  regarding  the  comparison  of  ECIT  and 
ACETEF  costs,  the  AF  included  estimated  costs  for  various  deployment  scenarios.  The 
AF  identified  6  major  categories  of  expense:  travel,  ground  transportation,  per  diem, 
equipment  shipments  (by  tactical  and  large),  and  aircraft  fuel  (also  by  tactical  and  large). 
These  costs  are  dependent  upon  several  key  factors  associated  with  test  moves. 
Speciflcally,  deployment  costs  are  driven  by  the  number  of  government  and  contractor 
people  (group  size),  number  (frequency)  and  duration  of  the  deployments,  the  type  of 
aircraft  being  us^  (tactical  versus  large)  which  largely  determines  the  quantity  of  materiel 
moved  and  aircraft  fuel  used.  The  AF  divided  the  factors  into  group  sizes  of  small, 
medium,  and  large;  into  durations  of  short,  intermediate,  and  long;  and  into  frequencies  of 
low,  medium,  and  high. 
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APPLYING  THE  AIR  FORCE  ESTIMATES 


We  implied  the  AF  deploymrat  cost  factors  for  various  deployments  of  AF  systems 
to  ACETEF  and  Navy  systems  to  ECIT  as  dictated  by  the  alternatives.  The  methodology 
we  employed  is  discussed  in  the  next  paragraph.  An  exception  is  the  deployment  estimates 
for  the  F-22  program  where  we  had  a  more  precise  deployment  schedule  and  a  speciHc 
Service  estimate.  In  addition  to  the  AF  estimate  of  the  cost  of  F-22  deployment,  we 
derived  a  less  conservative  and  less  costly  one  (the  base  case)  from  the  AF  estimate.  We 
also  developed  a  third  estimate  of  F-22  deplo3rment  costs  using  the  same  methodology  as 
for  other  systems.  All  three  are  shown  in  the  paper,  although  we  feel  the  base  case  is  the 
most  likely  of  the  three.  A  discussion  of  methodology  and  detail  used  in  estimating  the  cost 
of  F-22  deployments  follows  that  for  other  aircraft. 

All  Aircraft  Deployments  Other  Than  the  F-22 

Use  of  the  AF  deployment  factors  for  all  other  aircraft  systems  would  produce  a 
wide  range  of  possible  costs  for  various  d^loyment  situations  and  make  a  detailed  estimate 
by  deployed  system  for  all  systems  a  cumbersome  and  time-consuming  task  unless  a 
specific  individual  scenario  could  be  identified.  In  the  absence  of  this  information,  we 
elected  to  use  a  general  approach  that  would  approximate  an  "average"  deployment 

Discussions  with  AF  personnel  revealed  that  historically,  a  tactical  aircraft  typically 
would  use  a  small  group  of  people  while  a  large  aircraft  was  much  more  likely  to  use  a 
medium  or  even  a  large  group.  However,  given  the  projected  increased  demand  for  ISTF 
testing  during  the  next  several  years,  a  medium  group  size  would  be  reasonably 
representative  of  all  aircraft  systems. 

In  addition,  short  and  intermediate  durations  were  used  because  they  closely 
reflected  future  requirements.  A  requirement  was  judged  to  be  of  short  duration  if  less  than 
6  weeks.  In  the  AF  calculations,  a  short  duration  necessitates  only  one  trip.  A  requirement 
of  6  weeks  or  more  but  less  than  24  weeks  was  considered  to  be  of  intermediate  duration. 
The  AF  intermediale  duration  allows  for  one  round  trip  every  four  weeks. 

Table  D-1  summarizes  the  costs  of  deploying  various  AF  aircraft  to  the  ACETEF 
facility.  We  used  tire  latest  AF  requirements  for  ISTF  facilities  as  described  in  Section  V, 
which  totaled  127  weeks  per  year  with  an  additional  27  weeks  representing  a  20%  reserve. 
We  then  calculated  the  aimual  cost  by  aircraft  system  requirements  that  included  the  20% 
reserve  and  applied  one  of  two  basic  methods,  average  monthly  and  adjusted  short 
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For  the  average  monthly  method,  we  took  the  average  monthly  cost  for  either  a 
tactical  or  large  aircraft  (low  frequency  and  intermediate  duration),  divided  by  4  weeks  to 
obtain  the  average  weekly  cost  and  multiplied  by  the  number  of  weeks  in  the  annual 
requirement  to  produce  the  average  annual  cost  The  annual  cost  was  multiplied  by  20  to 
get  the  20-year  life-cycle  cost  estimate. 

In  the  adjusted  short  (duration)  method,  we  used  the  cost  category  data  for  the  short 
duration  as  follows:  Travel  +  Equipment  Shipments  -t-  Fuel  +  Appropriate  %  (based  on  the 
relative  number  of  weeks  using  the  2-week  period  as  the  base)  of  (Ground  transptntation  + 
Per  diem  +  Telephone/FAX). 

We  also  showed  the  average  weekly  costs  for  the  F-22  for  both  the  Hve  and  IS-year 
periods,  for  tactical  aircraft  and  for  large  aircraft 

F-22  Deployment  Estimates 

Table  D-2  shows  the  estimate  of  additional  costs  the  F-22  System  Program  Office 
(SPO)  believes  would  be  incurred  ^ould  it  be  required  to  use  the  ACETEF  facility  (see 
Appendix  F).  Hie  then-year  dollar  estimate  of  $357.6  million  equates  to  $308.9  million  in 
constant  fiscal  year  1992  budget  dollars^  The  only  breakout  furnished  by  the  SPO  is  as 
shown  in  the  tables.  "SIL-Cat  1"  includes  the  cost  of  basic  set-up,  support  equipment,  and 
instrumentation  for  two  avionics  lines  at  an  F-22  systems  integraticm  laboratory  (SIL)  to  be 
developed  at  Patuxent  NAS.  This  SIL  would  support  F-22  ACETEF  testing  only.  "SIL- 
Cat  2"  includes  "the  population  of  the  facility  with  actual  F-22  avionics  hardware  and 
associated  spares."  "SIL  Ops"  includes  the  cost  of  operating  the  SSL  at  Patuxent  with  75 
people.  "Flight  Ops"  is  the  SPO  estimate  of  the  added  costs  occasioned  by  the  loss  of  the 
use,  while  it  is  at  Patuxent,  of  one  F-22  test  aircraft  asset  from  the  planiwd  EdwanJs  flight 
test  program.  Costs  also  include  aircraft  spares,  and  operations  and  support  people  at 
Patuxent,  and  the  make-up  of  that  part  of  the  flight  test  program  (26  months)  that  the 
aircraft  at  ACETEF  would  have  flown  had  it  been  at  Edwards.  In  addition,  "a  25% 
allowance  for  regression  and  duplication  test  is  assumed  for  test  center-to-test  center 
correlation."  All  SPO  cost  estimates  include  15%  for  management  reserve. 

It  iq)pears  that  the  System  Program  Office  has  attempted  to  minimize  risk  by  treating 
all  non-chamber  ground  testing  at  ACETEF  as  an  add-on  that  would  have  to  be  replicated 


^  IDA  used  ooosiam  FY92  budget  yev  deflators  for  AF  RDTAE  as  puMisbed  by  OASD  (Canpiroller), 
17Jain»yl991. 


D-4 


elsewhere,  e.g.,  at  the  IFAST  at  Edwards.  Two  avionics  lines  would  be  required  at 
ACETEF.  There  are  no  o^setting  savings  for  SIL  work  done  at  ACETEF  that  would  not 
have  to  be  done  at  IFAST.  The  SPO  rationale  for  the  cost  estimate  of  flight  testing  states, 
"During  the  absence  of  the  test  asset,  it  is  assumed  that  other  avionics  test  assets  would 
make  up  some  of  tl^  lost  flights,  but  no  recoupment  of  the  test  time  was  considered  due  to 
the  risk  involved  in  that  assumption."  The  lost  time  (26  months)  of  aircraft  availability 
would  be  reaccomplished  in  FY  1998  and  FY  1999  with  the  previously  mentioned 
additional  25%  fm*  oenter-UK»nter  correlatitm. 


Table  D-2.  SPO  Eatimate  of  F>22  Deployment  Costs 


1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

Total 

Then- Year  DoOais 

SIL-Cail 

12.7 

13.0 

25.7 

SIL-Cat2 

22.4 

35.9 

31.0 

89.3 

SIL  Ops 

20.5 

21.2 

21.9 

22.5 

23.3 

6.0 

115.4 

Flight  Ops 

62.5 

64.7 

1272 

Total.  F-22 

22.4 

48.6 

44.0 

20.5 

21.2 

21.9 

85.0 

88.0 

6.0 

357.6 

FY1992Doll«s 

SIL-Catl 

12.3 

12.1 

24.4 

SIL-Cat2 

22.4 

34.6 

28.9 

85.9 

SIL  Ops 

18.5 

18.5 

18.5 

18.3 

18.4 

4.6 

96.7 

Flight  Ops 

50.9 

51.0 

101.9 

Total.  F-22 

22.4 

46.9 

41.0 

18.5 

18.5 

18.5 

69.3 

69.3 

4.6 

308.9 

Note:  Eitmutee  indode  SIL  wilb  two  aviooicf  liiiee  and  15%  management  reserve. 


IDA  believes  the  F-22  SPO  estimate  to  be  urmecessarily  conservative,  particularly 
since,  in  our  view,  the  experience  of  the  ACETEF  persoimel  and  the  capabilities  of  the 
ACETEF  facility,  are  unhkeiy  to  be  matched  anywhere  else  by  the  time  planned  for  the  start 
of  F-22  EC/avionics  ground  testing  in  FY199S,  even  should  the  ECIT  be  built 

The  IDA  base  case  is  based  on  die  SPO  deployment  estimate,  but  accepts  more  ridt 
As  shown  in  Table  D-3,  it  first  strips  out  the  15%  Management  Reserve  in  the  SPO 
estimate.  It  allows  a  SIL  at  ACETEF,  but  with  me  avionics  line.  Our  opinion  is  that  a  SIL 
collocated  with  tl»  ISTF  is  desirable  in  as  much  as  the  F-22  is  a  new  develqiment  with  the 
first  fully  integrated  EC/avionics  subsystems. 

Our  rough  estimate  of  the  effect  of  going  from  two  avionics  lines  to  one  line 
reduces  the  costs  of  the  SPO's  SIL  Category  1  and  SIL  Operations  by  a  third,  and  SIL 
Category  2  by  half.  For  flight  operations,  we  rejected  the  25%  add-on  for  cmta--to-center 
correlation,  and  assumed  half  the  flights  could  be  made  up  by  the  avionics  test  aircraft 
ranaining  at  Edwards.  This  results  in  a  60%  reduction  in  the  SPO  estimate  for  additional 
flight  operations.  The  estim^  of  the  total  cost  through  FY  2000  in  this  scenario  is  $143 
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million  constant  FY  1992  budget  dollars,  roughly  half  the  SPO  estimate.  The  detail  is 
shown  in  Table  E)-3. 


Tabi*  D>3.  Derivation  of  iDA  Eatimata  of  F-22  Dapioymant  Costa 


1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

Two  lines  (less  or 
witboia  1S%) 

Ihen- Years — SPO 

SIL-Catl 

11.0 

11.3 

22.3 

SIL-Cat2 

19.5 

31.2 

27.0 

77.7 

SIL  Ops 

17.8 

18.4 

19.0 

19.6 

20.3 

5.2 

100.3 

Flight  Ops 

54.3 

56.3 

110.6 

Total.  F-22 

FY  1992  $— SPO 

19.5 

42.3 

38.3 

17.8 

18.4 

19.0 

73.9 

76.5 

5.2 

311.0 

SIL-Catl 

10.7 

10.5 

21.2 

SIL<:at2 

19.5 

30.1 

25.1 

74.7 

SIL  Ops 

16.1 

16.1 

16.0 

15.9 

16.0 

4.0 

84.0 

Flight  Ops 

44.3 

44.3 

88.6 

Total,  F-22 

19.5 

40.8 

35.7 

16.1 

16.1 

16.0 

60.2 

60.3 

4.0 

268.6 

One  Line  (less  or 
without  15%) 

FY  1992$— IDA 

SIL-Catl 

7.1 

7.0 

14.1 

SIL-Cat2 

9.7 

15.1 

12.6 

37.4 

SIL  Ops 

10.7 

10.7 

10.7 

10.6 

10.6 

2.7 

56.0 

Flight  Ops 

20.4 

20.4 

40.8 

Total  F-22 

9.7 

22.2 

19.6 

10.7 

10.7 

10.7 

31.0 

31.0 

2.7 

148.3 

There  are  two  soisitivity  analyses.  One  uses  the  Air  Force  estimate  of  the  cost  of 
F-22  deployments,  as  shown  in  Table  D-2  in  constant  FY  1992  budget  year  dollars.  The 
second  assumes  that  the  F-22  could  use  the  same  technique  for  deployment  to  ACETEF  as 
was  used  by  other  AF  systems  such  as  the  F-16  and  the  EC-130,  i.e.,  a  standard 
deployment  with  no  SIL,  no  extra  spares,  etc.  The  estimated  cost  of  $1 1  million  over  the 
years  FY  1992  to  FY  2000  is  spread  as  shown  in  Table  D-4,  and  is  based  on  a  typical  AF 
large  group  (49  people),  deployment  costs  for  24  weeks  per  year  for  FY  1995  through 
FY  1999,  and  12  weeks  per  year  thereafter.  These  time  periods  include  20%  reserve.  The 
methodology  employed  is  die  same  as  diat  detailed  in  Section  V  of  the  paper. 


Table  D>4.  Minimum  F-22  Dapioymant  Coats 

2000  and 


1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

Bevcnd 

Weeks  Dq)loyed 

20.0 

20.0 

20.0 

20.0 

20.0 

10.0 

Wiih-^20% 

24.0 

24.0 

24.0 

24.0 

24.0 

12.0 

9  $91.4K/week 

0.2 

0.3 

1.9 

1.9 

1.9 

1.9 

1.9 

1.0 

TouJ.  FY  1992-2000  _ _ _ ILO 
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SENSITIVITY  ANALYSES: 

REQUIREMENTS,  COSTS,  AND  MEASURES  OF  MERIT 


This  appendix  provides  the  results  of  our  sensitivity  analyses  made  by  varying  F-22 
deployment  costs  and  changing  service  test  requirements.  First,  we  examine  the  base  case 
(100%  of  the  services'  electronic  combat  test  requirements  that  use  installed  systems  test 
facilities)  in  terms  of  needs  and  costs,  using  three  different  cost  estimates  for  F-22 
deployment  We  also  examine  the  base  case  with  varying  F-22  deployment  costs,  but 
considers  them  from  the  perspective  of  measures  of  merit  Next  we  show  cost  and  needs 
and  measures  of  merit  results  when  demand  is  changed  to  various  levels  between  50%  and 
150%  of  the  100%  base  case.  Finally,  we  summarize  the  overall  results. 

NEEDS  AND  COSTS 

The  columns  in  Tables  E- 1  through  E-6  (all  the  tables  appear  together  at  the  end  of 
the  appendix)  show  four  measures  of  merit:  (1)  capabilities  ("Percent  of  Needs  Met"),  (2) 
effectiveness  ("Increase  in  Needs  Met"),  (3)  "Additive  Costs",  and  (4)  measure  of 
effectiveness  ("Cost  per  Percent  of  Needs  Met").  In  each  of  these  tables,  the  results 
assume  a  demand  equal  to  100%  of  the  stated  requirement  The  various  rows  correspond 
to  the  current  situation  at  ACETEF  and  each  of  the  four  alternatives  in  the  study. 

Table  E- 1  shows  the  results  for  the  base  case  (which  uses  the  IDA  estimate  for  F-22 
deployment)  when  discounted  FY  1992  dollars  are  used.  Table  E-2  shows  the  same 
conditions  for  undiscounted  constant  FY  1992  dollars.  As  indicated  in  both  tables. 
Alternative  4  is  best  in  capability,  effectiveness,  and  measure  of  effectiveness,  and  poorest 
in  additive  cost  (Coincidentally  the  order  of  alternatives  also  shows  their  monotonically 
increasing  or  decreasing  alignment  in  this  Base  Case.)  If  one  looks  at  the  primary 
alternatives  of  interest  2  and  3,  the  cost  of  the  alternative  in  which  ECIT  is  developed  and 
used  (Alternative  3),  is  estimated  to  cost  7%  more  than  the  cost  of  developing  and  using  a 
large  anechoic  chamber  at  ACETEF  (Alternative  2).  However,  the  cost  per  percent  of 
additional  needs  met  with  ECIT  is  25%  less  than  the  ACETEF/LAC  alternative  due  to  the 
43%  increase  in  capability  with  the  ECIT  option. 
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Results  obtained  from  using  the  deployment  estimate  of  the  F-22  SPO,  rather  than 
that  of  IDA  are  shown  in  Tables  E-3  (discounted  constant  dollars)  and  E-4  (constant 
dollars).  Using  the  SPO  estimate  increases  the  costs  of  Alternatives  1  and  2  substantially. 
Now  the  least-cost  alternative  is  3  (ECIT),  and  it  would  be  less  costly  to  build  both  the 
LAC  and  ECIT  (Alternative  4)  than  to  build  the  LAC  alone  (Alternative  2).  (This  is 
because  the  F-22  deployment  costs  more  than  offset  the  cost  of  ECIT  but  would  be 
foregone  in  Alternative  4.)  Between  Alternatives  2  and  3,  the  alternative  with  ECIT  is  now 
75%  of  the  cost  of  that  with  LAC,  and  the  cost  per  percent  of  additional  needs  met  with 
ECrr  is  53%  of  that  with  LAC. 

The  results  using  the  minimal  deployment  estimates  are  shown  in  Tables  E-5  and 
E-6.  Even  though  the  cost  of  building  and  operating  both  ECIT  and  a  LAC  (Alternative  4) 
is  almost  twice  as  much  as  that  for  LAC  alone  (Alternative  2),  the  cost  per  percent  of  added 
needs  met  for  Alternative  4  is  slightly  better  than  for  Alternative  2. 

MEASURES  OF  MERIT 

If  one  assigns  a  1  to  the  "best"  alternative  and  a  "4"  to  the  worst  in  each  category, 
the  rankings  for  the  IDA  deployment  estimate  are  shown  in  Tables  E-7  and  E-8  for 
discounted  and  constant  dollars,  respectively.  (The  basic  data  correspond  to  Tables  E-1 
and  E-2,  respectively.)  In  this  set  of  tables,  the  capabilities  column  has  been  dropped,  but 
columns  showing  "Unquantifiable"  and  "Total”  have  been  added.  "Unquantifiable"  is  our 
subjective  overall  assessment  of  the  various  alternatives  in  areas  such  as:  potential  to  meet 
surges  in  workload,  meeting  current  test  philosophies,  avoiding  risk  in  weapons  system 
acquisition,  etc.  "Total"  is  the  sum  across  columns  for  each  alternative  and  is  based  upon 
equal  weight  assigned  to  each  category.  A  decision  maker  could  change  this  assumption 
and  weight  the  various  measures  of  merit  differently.  The  alternative  with  the  lowest  total 
score  is  best  For  the  base  case  shown  in  Tables  E-7  and  E-8,  Alternatives  4  and  3  are 
ranked  first  and  second,  respectively.  When  the  F-22  SPO  deployment  estimate  is 
substituted  for  that  of  IDA,  the  rankings  that  result  (Tables  E-9  and  E-10)  are  the  same  as  in 
the  base  case,  with  Alternatives  4  and  3  being  the  most  highly  rated. 

When  the  minimum  deployment  estimate  is  substituted  for  that  of  IDA,  Alternative 
4  remains  best  in  the  total  ranking,  but  Alternative  2,  that  of  building  a  large  anechoic 
chamber  adjacent  to  ACETEF,  ties  with  Alternative  3  if  discounted  dollars  are  used  (Table 
E-1 1),  and  is  superior  to  Alternative  3  in  constant  dollars  (Table  E-12). 
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RESULTS  FROM  VARYING  REQUIREMENTS 

Tables  E-13  through  E-24  are  in  the  same  order  as  those  described  previously,  but 
use  only  75%  of  the  stated  requirement.  For  the  IDA  anu  F-22  SPO  deployment  estimates 
(Tables  E-13  through  E-16,  E-19  through  E-22),  Alternative  3  (ECIT)  ranks  best  and 
Alternative  4  second  best.  This  is  a  reversal  of  their  order  with  100%  of  the  test 
requirement,  because  there  is  now  excess  capacity  in  Alternative  4.  When  the  minimum 
deployment  cost  estimate  is  used  with  75%  of  the  requirement  and  discounted  dollars 
(Tables  E-17  and  E-23),  the  best  ranking  is  with  Alternative  2  (build  large  chamber  at 
Patuxent)  followed  by  Alternative  1.  In  constant  dollars  (Tables  E-18  and  E-24),  the  best 
ranking  under  minimal  deployment  costs  and  75%  of  the  stated  test  requirement,  is 
Alternative  2  (no  new  chamber)  followed  by  a  tie  for  second  best  between  Alternatives  3 
and  4. 

Tables  E-25  through  E-36  present  data  similar  to  those  described  above  when  only 
50%  of  the  stated  requirement  is  used.  Only  with  the  F-22  SPO  deployment  estimate 
(Tables  E-27,  E-28,  E-33,  and  E-34),  does  Alternative  3  rank  best  and  Alternative  4  second 
best.  With  the  IDA  estimate  in  constant  dollars  and  50%  of  the  test  requirement  (Tables 
E-26  and  E-32),  and  with  the  IDA  estimate  discounted  (Tables  E-25  and  E-31),  the 
rankings  of  Alternatives  2  and  3  are  equivalent.  When  the  minimum  deployment  cost 
estimate  is  used  with  50%  of  the  requirement  (Tables  E-29,  E-30,  E-35  and  E-36),  the 
rankings  of  all  alternatives  are  equivalent  whether  or  not  constant  or  discounted  dollars  are 
used. 

For  increases  in  requirements  (Tables  E-37  through  E-48  at  125%  of  the  stated,  and 
E-49  through  E-60  at  150%)  the  relative  rank  order  does  not  change  from  comparable 
deployment  cost-estimating  assumptions  at  100%.  Alternative  4  (do  both)  is  best,  followed 
by  Alternative  3  (ECIT). 

SUMMARY 

Table  E-61  shows  in  a  single  table  all  the  preferred  alternatives  discussed  above. 
The  columns  show  ranking  results  of  the  various  sensitivity  excursions  in  requirements. 
The  rows  are  the  variations  in  deployment  cost,  with  two  measures  of  merit  for  each.  We 
believe  the  most  important  single  measure  of  merit  is  "dollars  per  additional  requirement 
met."  The  second  measure  of  merit  is  the  aggregated,  equal-weight  "total"  category 
discussed  above.  The  best  and  second-best  alternatives  for  each  of  these  two  categories  is 
shown  for  all  cases. 
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Table  E-61  shows  that  for  any  special  deployment  of  the  F-22,  Alternative  4 
dominates  under  requirements  of  100%  or  more.  Alternative  3  is  second  best  under  these 
full-requirement  options,  and  is  first  at  75%  of  the  stated  requirement.  We  think  these 
options  to  be  most  likely.  Alternative  2  is  best  for  a  mid-range  deployment  option  only  for 
50%  of  the  test  requirement.  If  a  minimum  deployment  cost  is  used.  Alternative  4  still 
dominates  for  cases  of  100%  or  more,  but  Alternative  2  is  second  best,  and  comes  to 
dominate  for  75%.  At  50%  of  the  requirement  with  minimum  deployment  cost,  the  first 
alternative  ranks  best  in  least  cost  per  unit  of  additional  requirement  met  even  though  no 
large  chamber  requirements  are  met;  however,  in  total  points  all  alternatives  are  equal. 
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Tabl*  E>61.  Prafarrad  Altarnativas 


Best.  Next  Best  Alternative 


50% 

75% 

100% 

125% 

150% 

With  IDA  Estimate,  Discounted  $ 

DoUafs/Requirement 

2,3 

3.4 

4.3 

4.3 

4.3 

Total 

2/3 

3/4 

4.3 

4.3 

4.3 

With  IDA  Estimate,  Constant  $ 

Dcdlats/Requirement 

2,1 

3.4 

4.3 

4.3 

4.3 

Total 

2/3 

3/4 

4,3 

4.3 

4.3 

With  F-22  Estimate,  Discounted  $ 

Dcrilvs/Requiiement 

3,4 

3.4 

4.3 

4,3 

4.3 

Total 

3/4 

3.4 

4.3 

4.3 

4.3 

MTitb  F-22  Estimate,  Constant  $ 

DoUars/Requiiement 

3,4 

3.4 

4.3 

4.3 

4.3 

Total 

3/4 

3/4 

4.3 

4.3 

4.3 

With  Minimum  Estimate,  Discounted  $ 

DoUars/Requhement 

1.2 

2.  1 

4.2 

4.2 

4.2 

Total 

1/2/3/4 

2,  3/4 

4.2/3 

4,2/3 

4,2/3 

With  Minimum  Estimate,  Constant  $ 

Dollars/Requhemoit 

1.2 

2.  1 

4.2 

4.2 

4.2 

Total 

1/2/3/4 

2.  3/4 

4.  2 

4.2 

4.2 

Notes:  A  diagonal  (/)  designates  a  tie.  Alternative  1,  provide  2.S  shifts  at  ACETEF; 
Alternative  2,  build  LAC  at  Patuxent  with  4-sbift  capacity;  Alternative  3,  build  ECIT  at 
Edwards  (and  Alternative  1  ACETEF);  and  Alternative  4,  build  LAC  and  ECTT  (Alternative  2  + 
Alternative  3). 
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APPENDIX  F 


SERVICE  COVER  LETTERS 


This  appendix  contains  copies  of  Navy  and  Air  Force  cover  letters  regarding 
ACETEF  issues  and  F-22  deployment  costs,  respectively.  The  Navy  letter  stresses  the 
advantages  of  ACETEF,  and  the  Air  Force  letter  provides  a  rationale  for  the  high  cost  of 
deploying  the  F-22  to  ACETEF  for  electronic  combat/avionics  ground  testing. 


F-1 


Ml  •(m  t* 


OEPARTMCNT  OF  THE  NAVY 
naval  AIK  tVSTCMS  COMMAND 
NAVAL  AIR  avSTKMS  COMMAND  HCAOOUARTCRS 
WASNINOTON.  DC  a03«t 

3900 

ser  NAWC-22B/018 
30  January  1992 

FfRST  ENDORSEMENT  on  Naval  Air  Warfare  Center  Aircraft 

Division  Itr  3900  Ser  SY02C/032  of 
13  Jan  92 

Conmander,  Naval  Air  Warfare  Center  / 

Deputy  Director  of  Defense,  Research  and  Enginee^n? 
(Test  and  Evaluation) ,  Office  of  the  Secretary  oy 
Defense,  Deputy  Under  the  Secretary  of  Test  and 
Evaluation,  Pentagon,  Washington,  D.C.  20301 
Chief  of  Naval  Operations  (OP-913) 

AIR  COMBAT  ENVIRONMENT  TEST  AND  EVALUATION  FACILITY 
(ACETEF) /ELECTRONIC  COMBAT  INTEGRATED  TEST  (ECIT) 

STUDY  COMMENTS 

1.  The  subject  letter  has  been  reviewed.  I  agree  with  both 
the  rationale  and  the  conclusions  in  the  letter  and  its 
supporting  docunentation.  The  consolidation  of  aircraft  and 
aircraft  systems  research  and  development  functions  from 
NAWCAD  Warminster  to  NAWCAD  Patuxent  provides  the 
opportunity  in  the  future  to  expand  the  use  of  ACETEF  to 
include  applied  research  and  early  development  tasks.  The 
facility  will  become  even  more  of  a  unique  national  asset. 
This  package  is  forwarded  with  my  strongest  recommendation 
that  the  Air  Combat  Environment  Test  and  Evaluation  Facility 
(ACETEF)  be  designated  as  the  primary  DoD  Integrated  Systems 
Test  Facility. 

2.  As  the  chartered  agent  of  Commander,  Naval  Air  Systems 
Command  (COMNAVAIRSYSCOM)  for  test  and  evaluation  facilities 
and  execution,  this  endorsement  constitutes  endorsement  by 
COMNAVAIRSYSCOM.  My  point  of  contact  is  Mr.  Tom  Metz  at 
DSN  286-7730  or  commercial  (703)  746-7730. 


From: 

To: 

Via: 
Subj : 


f  G.  H.  STROHSAHL 


Copy  to: 

NAWCAD,  Patuxent  River,  MD  (SY04,  SY02C) 
COMNAVAIRSYSCOM,  Washington  DC  (AIR-00,  05) 


OEPARTMEIUT  OF  THE  NAVY 

NAVAL  AM  WARf  are  CENTER 
AlflCRAFT  QMSiON 

PATUXENT  Rn/ER.  MARYLAND  cCS/O-ESCA 

3900 

Ser  SY02C/032 

j  ;  1  T  I9K 

From:  Commandor,  Naval  Air  Warfare  Canter  Aircraft  Division, 
Patxixent  River,  Karyland  20670-5304 
To:  Deputy  Director  of  Defense,  Research  and  Engineering 

(Test  and  Evaluation) ,  Office  of  the  Secretary  of 
Defense,  Deputy  Under  the  Secretary  of  Test  and  Evalua¬ 
tion,  Pentagon,  Washington,  D.C.  20301 
Via:  (1)  Commander,  Naval  Air  Warfare  Center  Headquarters 

(NAVAIRWARCEN-22) ,  Washington,  D.C.  20361-6000 
(2)  Chief  of  Naval  Operations  (OP-913},  Navy  Department, 
Washington,  D.C.  20350-2000 

Subj:  AIR  COMSAT  ENVIRONMENT  TEST  AND  EVALUATION  FACILITY 

(ACETEF) /ELECTRONIC  COMBAT  INTEGRATED  TEST  (ECIT)  STUDY 
COMMENTS 

Ref:  (a)  Draft  Memorandum  for  Chief  of  Naval  Operations  OP- 

91  from  D00R&£(T&E}  titled  Cost  Comparison  of  ECIT 
and  ACETEF 

(b)  ACETEF  Program  Meuiagement  Plan  of  15  April  1991 

End:  (1)  Supporting  Information  for  Institute  for  Defense 

Analysis  Study  of  ACSTSF/Large  Anechoic  Chamber  and 
ECIT 

1.  Congress  directed  the  Office  of  the  Secretary  of  Defense  to 
compare  the  costs  of  developing  the  ECIT  or  modifying  the 
ACETEF  to  support  Air  Force  programs.  Department  of  Defense 
Deputy  for  Research  and  Engineering  (Test  and  Evaluation) 
(DDDR6£(T&£) }  is  conducting  the  study.  'The  folloving  informa¬ 
tion,  as  requested  in  reference  (a) ,  is  provided  for  this 
study. 

2.  ACETEF  is  reco^ized  as  a  national  asset  and  is  the  world's 
leader  and  only  existing  (prognumed)  category  I  installed  sys¬ 
tems  test  facility  (ISTF)  existing  for  DT  and  OT  testing  of 
full  aircraft  systems.  ACETEF  with  its  existing  Hardvare-in- 
the-loop  (HIL) ,  man-in-the-loop  (MIL) ,  friendly  (blue) ,  poten¬ 
tial  enemy  (red)  and  commercial/third  world  (gray)  electronic 
combat  radiations,  targets  and  systems  stimulators  is  several 
years  ahead  of  the  ECIT  development  and  expertise.  Given 
today's  environment,  (i.e.,  rapidly  declining  DOD  budgets,  the 
recognition  of  the  Navy's  capability  by  the  emerging  Project 
Reliance  report  which  recognizes  the  Navy  as  lead  on  ISTF's, 
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plus  a  potantially  daclining  workload) ,  it  is  logical  to  con^ 
cluda  that  it  is  most  prudant  to  concentrata  af forts  on  ACETEF. 

3.  Tha  ACETEF  is  capabla  and  has  alraady  supportad  savaral  Air 
Forca  DT  and  OT  programs  such  as  tha  F-15E,  F-16,  and  EC-130 
aircraft.  Programs  such  as  tha  F-22  can  ba  accommodatad  in  tha 
axisting  ACETEF  anachoic  chambar  and  tha  B-1  and  B-2  in  tha 
shialdad  hangar  (with  tast  limitations)  until  tha  nav  anachoic 
chambar  is  built.  To  maat  this  incraasad  workload,  multipla 
shift  oparations  and  tha  concapt  of  oparations  as  dascribad  in 
paragraph  5  balow  ara  axpactad  to  ba  usad  and  hava  provan  to  ba 
highly  succassful.  Tha  Navy  and  its  ACETEF  ara  raady  to  assuma 
this  addad  rola  should  DOD  diract. 

4.  Tha  ganaral  raquiramant  of  an  ACETEF  typa  test  facility 
lias  in  six  areas:  tast  realism,  advanced  threat,  security, 
systems  integration,  interoperability,  and  cost.  Both  DT  and 
OT  tasting  must  ba  conducted  xindar  a  variety  of  conditions  that 
represent  conditions  in  tha  "real  world"  that  include  tha  ap¬ 
propriate  numbers,  types  and  proper  performance  of  threat, 
neutral  and  friendly  systems.  Both  MIL  and  MIL  systems  under 
tast  and  threat  systems  ara  required  to  assess  the  full  systems 
performance  including  tha  man-machine  interface.  This  along 
with  tha  number  of  special  access  programs  drive  tha  need  to 
perform  tasting  in  a  secure  tast  environment.  Tha  major  factor 
in  tha  need  for  ACETEF  typa  tasting  is  in  tha  areas  of  integra¬ 
tion  and  interoperability.  These  drive  both  tha  fidelity  and 
scope  of  system  and  stimulation  to  weapons  systems  or  multipla 
weapons  systems  in  the  latter  case.  Much  of  the  fidelity  and 
simultaneous  stimulation  cannot  be  performed  in  flight. 
Finally,  all  of  the  previous  factors  drive  the  cost  of  testing 
in  free  space  to  a  prohibitive  level,  without  the  use  of 
facilities  like  ACETEF  that  testing  would  not  be  able  to  be 
performed.  Enclosure  (1)  provides  specific  examples  of  Navy 
and  Air  Force  Test  requirements. 

5.  The  concept  of  operations  for  ACETEF  support  of  Air  Force 
programs  has  been  and  will  be  that  the  Air  Force  has  the  role 
of  test  conductor  while  the  Navy  hosts  and  provides  as  needed 
the  test  resource  support  along  with  engineering  support  if 
needed.  It  is  expected  that  a  program  P-22  would  use  the 
ACETEF  for  an  8  to  12  week  period  supported  by  the  Air  Force 
test  team,  system  contractors  etc. ,  conduct  the  tests  with  Navy 
support,  and  correct  problems  that  can  be  corrected  on  site. 
Other  hardware  and  software  corrective  actions  identified  in 
the  ACETEF  will  occur  at  the  prime  contractor/government 
development  site  with  the  data  collected  from  the  ACETEF. 
Usually,  programs  return  to  the  ACETEF  for  tests  after  correc- 
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tivtt  actions  hava  baan  takan.  Exaaplas  ineluda:  F-15,  F-16, 
EC-130  COMPASS  CALL,  F/A-18,  F-140,  BH-60,  AH-64,  atc. 

6.  Enclosura  (1)  providas  ansvara  to  tha  spacific  quastions 
diractad  to  tha  Navy.  Tha  ability  of  tha  ACETEF  to  carry  out 
tha  tasting  as  statad  in  tha  Sanata  Appropriations  Conaittaa 
intant  is  dapandant  on  tha  funding  laval  providad  in  ra far ones 
(b)  and  $60M  to  $70M  funding  for  tha  larga  anachoic  chaabar. 

7.  Our  point  of  contact  is  Mr.  Dan  Macona  at  DSN  326-6309  or 
conaarcial  301-737-6309. 


R .  PJ 
By  diraction 


OEPARTMCNT  OP  THE  AIR  FORCE 
HCAOouAirreRS  unitco  states  air  force 

WASHIROTON  OC 

1  7  APR  »92 

MEMORANDUM  FOR  DEPUTY  DIRECTOR  DEFENSE  RESEARCH  AND 

ENGINEERING  CTEST  AND  EVALUATION) 

SUBJECT:  Dstails  Supporting  F-22  *Diffirsntisl  Cost  EstiznAte  Rationale," 
ECnyACETEF  Comparison  •  INFORMATION  MEMORANDUM 

The  referenced  memorandum  requested  additional  data  in  support  of  the 
IDAded  ECn/ACETEF  Comparison  study.  The  requested  breakout  of  F-22  SPO 
estimated  costs  for  ACETBF  tasting  is  attached.  Ihe  plan  you  asked  for  describing 
infrasmiccure  capabilities  necessary  to  supporc  F-22  avionics  testing  is  the  F-22 
Avionics  Test  Concept  (ATC)  document.  Ihia  mtemal  program  office  document  was 
previously  provided  to  the  OSD  staff  as  a  reference  in  support  of  the  F-22  SPO's 
6  Jan  92  TEMP  submission. 

The  F-22  avionics  ssrstem  represents  one  of  the  most  challenging  Engineering 
and  Manufacturing  Development  test  and  evaluation  efforts  yet  undertaken  by  the 
DoD.  The  very  nature  of  the  F-22  design  requires  a  detailed,  rigorous,  incremental 
development  test  process  and  a  uniquely  configured  test  inffastructura  capable  of 
supporting  that  process.  A  collocated  installed  system  test  facility  is  an  essential 
tool  for  supporting  developmental  flight  test  of  the  F-22  or  any  aircraft  with  highly 
integrated  avionics.  We  believe  the  F-22  test  strategy  represents  the  optimum 
application  of  DoD  test  facilities  and  the  most  cost  effecdve  approach  for  supporting 
its  program  milestone  decisions. 

Requiring  the  F-22  to  test  at  Patuxent  lUvar  NAS  would  significantly  disrupt 
the  F-22  test  program.  It  would  add  at  least  $350  million  to  the  program  cost,  up  to 
8  additional  months  of  development  teat,  and  perhaps  most  importantly  add 
additional  technical  and  programmatic  risk,  b  order  to  acceptable 

davslopment  confidenca  levels,  the  SPO  would  have  to  substantially  replan, 
redirect,  and  duplicate  teat  assets,  tost  expertise,  and  test  personnel.  The  F-22  SPO 
would  have  to  deploy  and  operate  the  teat  aircraft  at  the  ACETEF  complex,  develop 
and  maintain  two  SILs,  pre-position  additional  aircraft  spares,  and  duplicate  both 
operational  and  support  personneL  Technically,  the  eha:^  would  require  the  SPO 
to  rely  more  on  contractor  development  testing  and  increase  the  likelihood  of  a 
fix-fly”  open  air  development  test. 

Also,  while  the  ACETEF  staff  has  a  "can-do*  attitude,  they  haven't  yet 
formally  committed  to  supporting  the  F-22  and  I  can't  help  but  be  concerned  that 
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th«  SPO  won't  be  fiUfy  lupportad  whan  tiuy  farizig  tbnr  raqiutraswat  fbr  26  aontha 
of  dediceted  "chimher  time*  (over  a  4»yeer  period  to  the  fadUty  maBager.  Im 
oonvineed  that  we  ere  getting  the  relationehip  between  program  and  teet 
iafraatructure  revegaed.  Adding  tedmieal  and  r^wg^*******  ■*<«»  gjalc  to  a  multibillion* 
dollar  program'e  teat  and  evaluation  etratagy  in  order  to  aeeommodate  a  daaire  to 

eonaolidete  twa  tipgiwd^  pawfaefea la loanaoeBO 

diaeemable  value  added  to  the  miadon  by  barring  BdT  upgradee. 

I  urge  you  and  your  etaff  to  e^qjport  ^  reinatatament  of  ftmda  for  die  ECrr 
progranu  We  really  must  have  the  appropriate  tools  to  conduct  disciplined  testing. 
The  DoD  doesn't  need  another  B-IB  test  and  evaluation  eaparienoe.  Pleaae  lot  me 
know  if  thare  ie  anything  elaa  you  need  in  developing  your  reeponse  to  the 
Congreea. 


lAteh 

F«22  Syatem  Program  Office  Cost 
Eitimata  at  the  ACETGF,  Mar  92. 
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F-22  SYSTEM  PROGRAM  OFFICE  COST  ESTIMATE 
FOR  THE  DUPLICATION  OF  REQUIRED  TEST  ASSETS  AT  THE 
AIR  COMBAT  ENVIRONMENT  TEST  AND  EVALUATION  FACILITY 


1.  Tlie  following  cost  esdmates  and  usodatedassampdons«cBpiovidediaiesposse  to  the  mqoixy 
of  the  maior  Dtotiain  costs  assodaied  with  having  a  geogtapiiieauy  septned  test  fkiUqr  tt  die  Air 
Combat  Eavsoameot  Test  and  EvahunonAciliqrCACErEF).  TaBseeostsareinidtfiratDthe 
•iiMrfypmgfMifwerf  costs  of  establishing  the  teynied  test  capabilities  at  the  F»22paiBaty  test  site, 
the  Air  Force  Flight  Test  CmterCAFFTQ.  The  total  costs  or  establiritina  the  additional  test 
e»p«hniiyatth^  ACgngpeflmpl«a«aiimfniriTRdbyfl3ealyearinTableL  T^  additional  costs 
w  into  thtee  te^eotegones  which  ate  the  establishment  of  a  Systems  humura  Liboiatoiy 
(SIL)  at  the  ACSTEPcomolex  the  deployment  of  a  flight  test  aircraft  to  ACfetar  for  an  extended 
pedod.  and  the  operation  and  support  m  ^  SIL  and  the  aheaft  while  as  die  ACE7EF  coaplex. 
fte  breakdown  M  each  of  these  cost  caggones  are  discosaed  in  subsequent  paragraphs. 


FVW 

Bca 

FV94 

Bffii 

Em 

214 

48.6 

44.0 

20.5 

112 

FY97 

ssa 

Em 

FVPP 

TDTAL 

21.9 

8S.0 

88.0 

6.0 

357.6 

2.  The  following  assumptions  and  cost  breakdown  is  in  support  of  establishing  a  Systems 
Ihtegradon  Laboratory  (^)  and  a  basic  avionics  test  and  anomaly  hxvenigaiion  capebiBqf  tf  the 
ACsTEFcompl^  inidal  assumption  is  that  facility  space  of  15.000«|uare»t  is  available  to 

the  F-22  pngtam  at  the  ACSTEF  eompia  with  suffldent  soiw  and  cooling  and  open>air  lange 
access  to  support  the  SIL  operating  tequiremeata  Should  ouiefimilitymgtades  be  requmd  to 
meet  these  rgquireT"w«y,  fh^enaaMenflteQHsidatBdlBtfaisfittimetB.  The  costs  fisr  establishing 
the  SIL  are  broken  down  into  two  categttiea  Caiegoiy  IwfaidiindndesifaBeosaofbesie  ftexiiqr 
seora ,  support  equipmeoL  md  instrameniidon.  and  Cai^oty  2  which  indsdes  the  popuisnon  of 
the  BcilitywiAscmsl  F-22  avionics  hardware  and  sssocisted  spares.  These  costs  ate  detailed  by 
fiscal  yeer  in  Table  1 


FW2 

FY93 

FY94 

TOTAL 

Category  1 

117 

13.0 

25.7 

Category  2 

SIL  Total 

214 

35.9 

31.0 

M3 

115.0 

Tthte  r  Thtemtion  Laborattarv  Com  (TV  S  in 


3.  Use  following  issnmpdions  ind  con  breakdown  are  in  rapport  of  depbyingan  tvionies  flight 
tut  aircraft  to  the  ACSItfflraili^  for  avimics  ground  and  chamber  testing.  Tliisrequitu 
additional  aircraft  apyes  to  be  poraiooed  at  Paruxent  River  Naval  Air  Station  (NAS),  aa  well  aa,  the 
duplication  of  operatiott  and  rapport  petsranel  for  the  maintenance  and  aopportrftte  aircraft  The 
achedole  of  airborne  avioniu  ten  at  tu  primary  test  location  would  be  impaeaed  by  the  lost  teat 
asset  while  it  is  deployed.  Doting  the  absence  of  the  test  asset,  it  is  assui^  that  other  avioniu  test 
assets  would  make  up  some  of  the  lost  flights,  but  BO  tecoupmera  of  the  test  time  wu  consideaed 
due  10  the  risk  InvcitM  in  that  assumption.  The  lost  test  dr^  amounts  to  26  months  of  airctaft 
availaUlity  during  the  avionks  flight  lest  develMnent  phase.  This  lost  time  would  be 
reaccomplished  in  FY9i  and  FY99  and  a  23%  atlowanu  for  regression  and  duoiiurion  test  is 
assumed  for  test  eeate^tD-lest  center  correlation.  The  cost  estimatu  assume  2oaitcnft  flying 
hours  per  agnth  and  the  cosa  are  calculated  based  on  labor  hours  assuming  1,750  labor  hours  per 
flight  test  hour  at  S70  per  hour.  These  costs  are  deoiled  by  fiscal  yw  in  Table  3. 

FV9g  FV90  TOfTAL 

Aircraft  Fnght  Test  Cbso  62J  64.7  127.2 

Tahlel.  Airet^  Flight  Test  CoflarTYS  in  mfllioM) 


4.  The  following  assumptiou  and  cost  breakdown  are  in  support  of  the  operation  of  the  SS,  at  the 
ACETEFeomptex.  The  SILregutru  100  contractor  person^  for  the  basrefira£hiy  operation  and 
test  and  evaluation  tasks.  Based  on  the  reduced  scope  of  effort  to  be  accomplished  at  the  AGETEF 
complex,  only  75%  or  75  personnei  win  be  required.  The  fikcility  will  be  operating  for  63  months  at 
160  ixiun  a  month  (huing  fiscal  years  FY95  throuitit  FYQQ.  The  basic  pay  rate  is  assumed  to  be 
S 100  per  hour  in  BY90S  which  caiculata  to  SUM  per  month.  These  costs  are  then  convened  to 
TYS  anti  are  detailed  in  Table  4. 

FV95  £226  nSL  ET2a  Fvm  total 

Sn.  Operating  20J  21.2  21.9  22J  23  J  6.0  115.4 

Costs 


Tihle  a.  STL  Oerxiinv  Cam  rTV  S  in  trnTKoM^ 
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ADDITIONAL  ISSUES 

UNIQUE  CAPABILITIES  OF  INSTALLED  SYSTEMS  TEST  FACILITIES 

This  section  summarizes  some  of  the  key  points  made  by  the  Navy  based  on  their 
ACETEF  experience. 

The  most  "up  to  date"  hardware  and  software  conHguration  available  for 
engineering  test  is  the  hardware  and  software  that  is  physically  installed  in  an  aircraft.  The 
"spread  bench"  configuration  often  is  different.  The  "SIL"  is  used  principally  by  the 
contractor  to  physically  integrate  the  system — not  to  support  ISTF  or  flight  test 

The  Navy  views  such  integration  facilities  as  the  SIL  as  being  suited  for  tests  where 
one  wants  to  get  "in  the  box."  Hardware-in-the-loop  (HTTL)  facilities  are  used  for  testing 
"between  the  boxes."  ISTFs  are  used  for  testing  of  system-level  responses.  (Note  that  the 
IF  AST  falls  more  within  the  Reliance  definition  of  hardware-in-the-loop  than  integration 
facility.) 

The  Navy  outlined  five  classes  of  testing  where  an  ISTF  provides  the  ability  to  do 
something  not  possible  in  an  integration  facility: 

•  Tests  that  require  a  level  of  stimulation  greater  than  that  likely  to  be  found  in 
other  less  C2q)able  facilities  such  as  SILs  and  HTTL  facilities. 

•  Tests  that  require  the  test  article  to  be  "installed"  in  order  to  properly  evaluate 
the  potential  effects  of  the  airframe  cm  system  performance. 

•  Tests  that  require  the  system's  radiating  elements  to  operate  in  an  "in  flight" 
ctmfiguration. 

•  Tempest  tests  of  specific  hardware  installations  (as  opposed  to  designs). 

•  Tests  that  require  absolute  security  of  RF  signals,  either  threat  or  friendly. 

Examples  for  these  classes  of  tests  are  described  in  the  paragraphs  that  follow. 

Case  1 — Tests  requiring  a  higher  level  of  stimulation  capability  than  is 
likely  to  be  available  at  other  than  ISTF  facilities.  A  specific  example  is  the  test  of  the 
ALO-149  Communication  Intercept  Receiver  for  the  EA-6B  AVCAP  program.  The  system 
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had  been  bench  tested  successfully  by  the  manufacturer,  and  had  passed  its  Acceptance 
Test  Procedures  in  an  HWIL  facility  at  PMTC  over  a  3-  to  6-month  test  program.  The  first 
aircraft  with  an  installed  system  then  went  to  the  ACETEF  for  testing.  During  the  first 
wedc  of  testing  in  the  chamber,  the  system  failed  when  subjected  to  a  dense  communicatitm 
environment  (Buffers  internal  to  the  system  became  saturated,  and  the  processor  locked 
up.)  In  tteory,  had  the  HWIL  facility  at  PMTC  been  adequately  facilitated  with  high- 
density  stimulators,  this  problem  could  have  been  found  there. 

However,  because  each  program  tends  to  have  (and  need)  its  own  SIL  and  HWIL, 
such  an  investment  would  need  to  be  repeated  at  every  SIL  and  HWIL  in  the  Navy.  This  is 
not  the  Navy's  approach.  Rather,  the  Navy  has  concentrated  capability  at  ACETEF.  It  is 
not  the  anechoic  chamber,  per  se,  or  even  the  concept  of  an  ISTF,  that  solves  the  Case  I 
type  problem,  as  much  as  it  is  a  strategy  that  creates  a  limited  number  of  high-value 
facilities,  by  putting  the  high-capability  simulators  and  stimulators  at  an  ISTF. 

Case  2 — ^Tesis  wherein  the  interaction  of  the  airframe  and  the  test  article, 
in  a  radiating  environment,  are  of  interest  The  High  Speed  Anti-radiation  Missile  (HARM) 
is  carried  on  the  E/A-6  aircraft  Testing  in  the  ACETEF  revealed  a  directional  problem — 
the  threat  signal  reflected  off  the  A-6  airframe  into  the  seeker  of  the  HARM.  This  caused 
threats  that  were  up  to  90  degrees  off  the  nose  to  appear  to  be  on  the  nose.  This  problem 
could  only  be  identified  in  a  facility  that  permits  the  test  article  to  be  "installed"  and  the 
threat  to  be  radiating.  A  SIL  and  HWIL  would  not  (and  did  not)  identify  this  class  of 
problem.  The  ACblEF  identified  it  quickly. 

Case  3— Tests  requiring  that  the  system  be  radiating  in  an  "in  flight" 
fashion.  The  F-14D  program  involved  integration  of  a  GFE  Electronic  Warfare  system  in 
the  airplane.  The  system  consisted  of  the  ALR-67  Radar  Warning  Receiver  (RWR),  the 
ALQ-165  Airborne  Self  Protection  Jammer  (ASPJ),  and  the  ALE-39  Chaff  Dispenser.  The 
aircraft  itself  was  equipped  with  the  APF-7  Attack  Radar.  The  system  was  tested  in  the 
contractor's  ISL  and  determined  to  be  operating  correctly.  NAVAIR  directed  that 
additional  testing  be  performed  by  Grumman  at  ACETEF.  NATC  provided  the  facility, 
"the  electrons,"  and  general  oversight  for  safety  and  reasonableness  of  test  methodology. 
The  contractor  conducted  the  test,  which  ran  from  1  January  through  31  March  1990.  The 
testing  revealed  22  major  problems,  17  of  which  were  fixed  during  the  period  of  the  test 
Basically,  by  testing  in  a  facility  that  put  all  the  subsystems  into  their  installed 
configuration,  while  radiating,  and  in  a  threat  environment,  the  Navy  was  able  to 
demonstrate  that  the  attack  radar  was  jammed  by  the  ASPJ,  and  that  numerous  other 
"software  timing"  problems  existed.  These  problems  were  not  apparent  in  the  spread 
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bench  layout  and  geometry  of  the  SIL,  where  radiating  elements  were  not  operating  in  a 
normal,  "installed"  fashion.  These  sorts  of  tests  require  an  ISTF. 

Case  4 — Tempest  tests  that  evaluate  the  adequacy  of  speciHc 
inftttllatiftnie.  Certain  Navy  systems  have  highly  classified  data  available  within  the 
aircraft  in  a  variety  of  frequencies.  Although  the  inherent  design  of  such  systems 
theoretically  prevents  inadvertent  emission  of  such  data,  the  actual  manufacturing  and 
assembly  process  may  not  afford  the  design-level  protectitm.  Testing  an  aircraft  in  an  ISTF 
can  provide  reasonable  assurance  that  the  aircraft  in  fact  protects  the  classified  signals 
present  within  it  This  sort  of  testing  is  only  possible  in  the  electromagnetically  "quiet" 
environment  available  in  an  ISTF's  anechoic  chamber,  where  adequate  simulation  and 
stimulation  are  also  available.  For  classification  reasons,  the  Navy  was  not  willing  to 
address  specific  aircraft  problems  in  this  arena. 

Case  5 — Testing  requiring  signal  security.  Tests  often  require  a  secure 
environment  in  which  the  threat  emitter  and/or  the  system  under  threat  can  radiate.  Only  an 
ISTF  with  its  anechoic  chamber  provides  this  capability.  We  did  not  discuss  specific 
examples  with  the  Navy,  but  one  such  use  is  tte  testing  of  signal  war  reserve  modes. 

FREE-SPACE,  FAR-FIELD  TESTING 

The  issues  of  anechoic  chamber  size  and  method  of  system  stimulation  (signal 
injection  or  use  of  "antenna  hats"  versus  "free-space,  far-field  radiation")  have  been 
mentioned  several  times  in  this  pq)er  as  technical  issues.  It  is  beyond  the  scope  of  this 
paper  to  explain  these  issues  in  detailed  technical  terms.  The  discussion  below  attempts  to 
provide  some  insight 

Two  approaches  are  theoretically  available  for  stimulating  a  system  under  test 
(SUT)  in  an  anechoic  chamber.  These  are  signal  injectitm  and  "antenna  hats";  ot  free-space 
radiation.  In  an  ideal  test  chamber,  both  would  jnobably  be  used. 

Signal  injection  and  antenna  hats  involve  delivering  a  signal  from  a  stimulate  (e.g., 
from  a  threat  simulator)  to  the  SUT  via  cables,  where  the  signal  is  radiated  directly  into  the 
receiving  antennas  of  the  SUT  by  means  of  antenna  hats;  or  is  injected  directly  into  the 
SUTs  signal  processing  circuitry  without  passing  through  the  antenna. 

Free-space  radiatitm  means  that  the  radiating  stimulator  delivers  a  signal  to  die  SUT 
by  actually  transmitting  into  the  chamber,  with  the  SUTs  antennas  then  receiving  the 
signal. 
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The  implication  of  "far-field"  is  that  the  distance  between  the  SUT  and  the  radiating 
stimulator  must  be  sufficient  that  the  SUT  is  not  subject  to  phenomena  that  are  so  highly 
localized  around  the  transmitter  that  an  actual  system  in  flight  would  not  be  affected  by 
those  phenomena. 

The  theoretical  benefits  of  "free-space,  far-field  radiation"  are  that  it  tests  the 
antennas  and  any  signal  processing  circuitry  imbedded  in  the  antennas,  and  that  it  facilitates 
evaluation  of  phase  measurement  perfcMinance  of  the  SUT.  On  the  other  hand,  it  creates  a 
more  complex  overall  test  simulation  environment,  and  there  is  significant  debate  within  the 
test  ccxnmunity  regarding  the  physical  characteristics  an  anechoic  chamber  must  possess  to 
permit  fiee-space,  far-field  testing. 

Notwithstanding  the  complexity  of  the  debate,  far-field  testing  (and  its  presumed 
large  chamber  requirement)  had  little  bearing  on  our  recommendation  to  proceed  with  the 
ECrr  at  Edwards  AFB.  Our  case  can  be  summarized  as  follows: 

•  To  satisfy  the  ISTF  requirements,  more  chamber  capacity  and  a  large  chamber 
capability  are  needed. 

•  The  additional  large  chamber  ISTF  edacity  can  be  accomplished  by  adding  a 
large  chamber  to  ACETEF  or  by  adding  facilities  (simulation  and  stimulation 
labs,  and  command  and  control)  to  the  chamber  at  Edwards.  Our  cost  analysis 
indicates  that  ECTT  is  move  cost  effective;  at  the  wotldoad  levels  we  believe  are 
likely  and  with  investment,  deployment  and  operating  costs  we  believe  are 
most  probable. 

•  If  OSD  decides  to  pursue  this  option,  there  is  the  added  potential  benefit  that 
the  ECrr  is  more  likely  to  permit  evaluation  of  free-^ace,  far-field  techniques, 
which  may  or  may  not  work,  and  may  or  may  not  prove  to  be  worthwhile  if 
they  do  work. 

While  it  would  be  gratifying  to  be  able  to  resolve  the  technical  issues  identified 
above,  even  technical  experts  in  the  field  have  differences  of  opinion  cm  the  subject  We  do 
not  believe  the  issue  will  be  fully  resolved  soon.  On  the  other  hand,  we  do  not  believe  that 
the  issue  is  crucial  to  the  ECTT  versus  ACETEF  decision. 

The  pages  that  follow  contain  materials  prepared  by  the  Air  Force  that  elaborate  on 
this  issue. 
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POLARIZATION  AND  PHASE  CONSIDERATIONS 
FOR  INSTALLED  SYSTEMS  TESTING 


QUESTION 

Why  are  the  polarization  and  phase  characteristics  of  electromagnetic  emissions 
important  for  installed  avionics  systems  testing? 

ISSUE 

This  paper  discusses  the  requirements  for  installed  systems  test  using  bi-directional 
(SUT-to  threat  and  threat-to-SUT)  free-space  radiation  and  signal  injection  techniques. 

IMPLICATIONS 

The  test  technique,  free  space  radiation  or  signal  injection,  is  a  driving  requirement 
on  test  chamber  size  and  its  anechoic  characteristics.  If  signal  injection  is  the  only 
stimulation  technique  is  the  only  stimulation  technique  required,  then  the  chamber  size  and 
its  anechoic  characteristics  are  not  critical.  If  free-space  radiation  is  required,  then  the 
chamber  must  not  only  have  good  anechoic  characteristics,  but  it  must  also  be  sized  to 
provide  a  "quiet  zone"  large  enough  to  accommodate  the  largest  article  to  be  tested. 

DISCUSSION 

1.0  This  paper  discusses  three  issues:  (1)  electromagnetic  emission  polarization 
and  phase  in  electronic  combat  system  operation;  (2)  identiflcation  of  some  avionics/EC 
systems  which  use  sigiuil  polarization  and  phase  to  function  properly;  and  (3)  importance 
of  testing  these  type  systems  through  both  signal  injection  and  free  space  radiation. 

1.1  Electromagnetic  energy,  when  transmitted  through  space,  has  important 
properties  for  avionics  and  their  imbedded  electronic  combat  functions.  These  properties 
are  the  frequency,  amplitude,  polarization,  and  phase  of  the  energy  impinging  on  the 
system  under  test  Because  of  their  importance  in  evaluating  ECM  and  ECCM  functions, 
the  properties  addressed  specifically  in  this  paper  ate  polarization  and  phase. 

1.2  Polarization  and  phase  measurements  ate  used  for  determining  the  direction  of 
arrival  (DOA)  of  energy  impinging  on  the  system  under  test  (SUT)  and  for  assessing  radar 
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apeiture/scan  controller  operation  under  threat  exploitation.  Most  modem  and  future 
aircraft  weapon  systans  being  developed  by  the  DoD  will  have  electronic  combat  functions 
that  use  transmitted  and  received  signal  polarization  and  phase  characteristics.  Some 
ground,  air,  and  sea  threat  systems  exploit  signal  polarization  and  phase  to  defeat  the 
weapon. 

1.2.1  ECCM  equipment  (double  jammers,  sideline  blanking  jammers,  and  stand¬ 
off/escort/stand-forward  jammers)  take  advantage  of  the  E(XM  functions  inside  search, 
track-while-scan  (TWS),  and  track  radars.  The  techniques  which  these  systems  employ  are 
called  cross-polarization,  cross-blinking,  and  terrain  bounce/bounce  jam.  If  these 
techniques  are  successful,  then  the  victim  sensor's  ECCM  functions  are  impaired.  This 
occurs  within  the  victim's  radar  which  is  then  unable  to  defend  against  enemy  jamming 
while  targeting  and  firing  wei^ns. 

1.3  One  of  the  objectives  of  testing  electronic  combat  systems  is  to  effectively 
replicate  the  typical  combat  situations  that  these  systems  encounter.  The  installed  systems 
test  facilities,  like  the  Benefield  Anechoic  Chamber  (BAF)  should  be  able  to  replicate 
realistic  combat  situations  in  a  secure  and  electromagnetically  quiet  location.  However,  the 
challenge  to  replicate  the  many  electromagnet^  emission  characteristics  for  every  emitter, 
on  every  player,  and  within  a  combat  scenario  is  dependent  upon  electromagnetic 
characteristics  of  the  anechoic  chamber  when  using  free-space  propagation  or  the  digital 
representation  when  using  signal  injection. 

1.3.1  If  the  testing  technique  is  signal  injection  behind  the  antenna/{q)erture  and 
scan  controller  equipment,  then  the  electromagnetic  energy  wavefront  polarization  and 
phase,  as  well  as  the  antenna  characteristics,  must  be  digitally  calculated  and  included  in  the 
injected  signal  In  this  situation  the  antenna/aperturc  and  scan  controller  are  untestable  and 
jamming/countermeasure  techniques,  which  attack  the  vulnerabilities  of  this  equipment, 
cannot  be  evaluated  except  by  digital  simulation  and  analysis  of  simulation  results.  Digital 
simulation  will  be  further  complicated  by  the  next  generation  of  weapon  system  antennas, 
which  will  contain  active  components.  Moreover,  accurate  system  performance  and 
effectiveness  of  the  SUT  cannot  be  detennined,  confidence  in  system  operational 
performance  is  reduced,  and  development  risk  is  increased.  The  chamber's  anechoic 
properties  are  not  a  factor  in  these  types  of  test;  and,  generally  speaking,  the  chamber  size, 
if  large  enough  to  accommodate  die  SUT,  is  not  important 

1.3.2  On  the  other  hand  if  the  total  weapon's  effectiveness  is  to  be  evaluated,  then 
the  system  must  also  be  tested  by  free-space  radiation.  In  this  case  the  SUT 
antennaAqierture  and  scan  controller  must  be  subjected  to  the  electromagnetic  energy  source 
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so  that  the  energy  wavefronts  are  reinesentative  of  those  that  occur  ntumally  during  combat 
situations.  Since  the  antmina/qtetture'  and  scan  controller  are  included  in  the  testing,  their 
effects  on  the  SUTs  performance  is  an  integral  consideration  during  the  testing.  The 
whole  system  is  being  tested,  confidence  in  system  performance  is  increased,  and 
development  risk  is  reduced. 

Testing  by  signal  injection  is  less  demanding  on  anechoic  chamber  performance, 
but  has  limited  utility  for  SUTs  that  employ  si^ial  polarization  and  phase  characteristics. 
Free-space  radiation  {Hovides  greater  test  realism  but  requires  a  chamber  sufficiently  large 
to  create  an  electrcnnagnetic  quiet  zone  accommodating  the  SUT  (see  "Anechoic  Chamber 
Sizing  Considoations  for  Installed  Systems  Te^g”).  Depending  upon  the  type  of  test  to 
be  performed,  the  fidelity  and  number  of  threats  employed,  and  the  required  equipment  to 
be  tested,  injectitm  and  free-q»ce  are  both  desirable  test  implementatitm  techniques.  Each 
of  diese  implementation  techniques  has  significant  purpose  and  provides  a  different  level  of 
realism.  The  use  of  only  one  technique  limits  a  chamber's  usefulness  and  produces  a 
limited  set  of  test  results.  This  means  less  system  conEdence  and  higher  development  risk. 
The  optimal  implementation  is  the  complementary  and  synergistic  use  of  both 
electromagnetic  emission  techniques  to  provide  realistic  environments  sufficient  to  meet 
tests  objectives. 
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ANECHOIC  CHAMBER  SIZING  CONSIDERATIONS 
FOR  INSTALLED  SYSTEMS  TESTING 


QUESTION 

What  is  the  minimum  required  anechoic  chamber  size  to  meet  Air  Force  installed 
system  test  (1ST)  requirements? 

ISSUE 

Chamber  size  is  a  critical  factor  in  the  ability  to  accommodate  testing  with  bi¬ 
directional  free-space  radiation  of  electromagnetic  signals  emanating  from  both  the  system 
under  test  (SUT)  and  the  threats  being  tested  against  Minimum  quiet  zone  size  and  energy 
source  minimum  distances  from  the  SUT  are  critical  to  accommodate  Air  Force  test 
requirements. 

IMPLICATIONS 

If  the  anechoic  chamber  is  not  sufficitmtly  large,  then  all  Air  Force  test  requirements 
may  not  be  accommodated. 

DISCUSSION 

1.0  Summary  of  Analysis:  Attached  is  part  of  the  analysis.  Technical  Plan  for  the 
Design/Build  Hiase  of  the  Air  Force  Anechoic  Facility",  accomplished  by  the  Air  Force, 
under  contract  to  Rockwell  International,  to  design  and  build  the  Benefield  Anechoic 
Chamber  at  Edwards  AFB.  Below  is  a  quick  summary  of  the  considerations  that  were 
made  by  the  Air  Force  to  arrive  at  a  design  that  was  the  minimum  necessary  to 
ttxommodate  the  Air  Force  requirement,  not  just  for  the  B-IB,  but  also  for  other  large  Air 
Force  weapon  systems,  such  as  the  B-S2,  C-17,  EC-130,  B-2. 

•  Dimenaons  of  the  B-IB:  Wing  Span  137  ft.  Length  147  ft.  Height  34  ft 

•  Minimum  clearance  between  the  chamber  RFabsorbo*  surface  and  the  aircraft 
under  test  is  IS  fL  (This  provides  a  free-space  loss  of  from  52  dB  at  1  GHz  to 
77  dB  at  18  GHz  when  combined  with  the  RF  absorber  loss,  a  minimum 
dhamber  isolation  in  die  mtt  zone  of  over  100  dB  will  be  attained.) 
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•  In  order  to  properly  test  the  aft  section  of  the  B- IB,  a  minimum  separation  of 
75  ft  between  the  chamber  RF  absorber  surface  and  the  tail  radome  is  required. 
This  means  that  the  aircraft  will  not  be  centered  in  a  minimum  sized  chamba*. 

•  B-52  and  C- 17  Design  Considerations 

When  all  of  the  above  factors  are  considered,  it  can  be  shown  that  the  minimum 
chamber  requirements  (including  RAM  dimensions)  for  the  B-IB  are  241  ft  long  X  171  ft 
wide  X  68  ft  high.  When  the  B-S2  and  C>17  requirements  are  considered  in  conjunction 
with  tire  B-IB  the  minimum  accqptable  dim^ions  become  260  ft  long  X  240  ft  wide  X  70 
ft  high. 

2.0  Power  Dissipation  Considerations:  The  attached  Technical  Plan  indicates  that 
the  high-powered  B-IB  offensive  radar  system  is  capable  of  operating  in  a  chamber  (with 
dimensions  as  given  above)  with  a  low  power  absorber  material  capable  of  dissipating  only 
2  watts  per  square  iireh.  Since  absorber  material  either  has  high  power  dissipation  edacity 
or  high  absorption  but  not  both,  a  similar  evaluation  must  be  performed  on  any  irew  design 
to  insure  that  the  chamber  is  large  enough  to  use  the  low  power  high  absorption  material, 
otherwise  the  more  expensive  high  power  material  must  be  used. 

3.0  AdditicHial  volume  provided  by  a  chamber  that  will  accommodate  large  as  well 
as  small  SUTs  will  result  in  more  valid  test  results  by  providing  longer  propagation  paths 
for  multipath  error  agnals  and  as  a  consequence  lower  corruption  of  actual  test  signals  and 
installed  avionics  systems  re^nse. 

CONCLUSIONS 

Air  Force  test  requirements  for  an  installed  system  test  facility  includes  the 
requirements  to  test  both  by  signal  injection  and  by  bi-directional  firee-^ace  radiation  (see 
"Polarization  and  Phase  Considerations  for  Installed  Systems  Testing”).  This  requirement, 
as  well  as  other  requirements  dictated  by  power  and  signal  handling  capacities,  dictate  the 
need  for  an  anechoic  chamber  of  sufficient  size  to  accommodate  all  Air  Force  weapon 
systems,  small,  large,  and  multiple  aircraft 

Based  on  Air  Force  analysis,  the  smallest  chamber  that  will  support  the  Air  Force 
requirement  is  an  airechoic  chamber  with  the  inside  dimensions  at  least  260  ft  long  X  240  ft 
wide  X  70  ft  high. 
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POTENTIAL  FOR  USE  OF  ANECHOIC  CHAMBER  FACILITY  AT 
GRUMMAN  AIRCRAFT,  CALVERTON,  NY 

The  Navy-owned  chamber  at  Calveiton  is  roughly  the  size  of  the  ACETEF 
chamber,  with  very  limited  supporting  laboratories.  Providing  it  with  the  capability  to 
support  testing  similar  to  the  existing  ACETEF  capability  would  require  an  investment 
comparable  to  that  which  IDA  estimates  for  the  ECIT — and  would  still  provide  no 
ctq)ability  for  testing  of  large  aircraft 


ACETEF 

Grumman 

Size 

60’xl00*x40*(h) 

87*  X  87’ X  42(h) 

Door 

40’x20’(h) 

80’x40’(h) 

Crane 

30  ton 

30  ton 

Frequency  Range 

O.Slo94GHz 

0.06  to  18  GHz 

IOC 

1983 

1969 

Uses 

EMC.  EW.  EC  Aircraft  Systems  Integration 

EMC  EMI,  Radiation  Hazard 

n-i- Cat  I  iriiBined 

m 

Limitations  of  the  Calverton  chamber. 

•  The  chamber  at  Grumman  cannot  handle  the  "large  chamber”  requirements. 

•  It  possesses  very  limited  stimulation  and  no  simulation  capability.  It  can 
generate  dozens  of  relatively  non-complex  signals  compared  to  hundreds  of 
relatively  high-fidelity  threats  available  in  ACETEF. 

•  There  is  no  man-in-the-loop  ctqiability. 

•  It  cannot  provide  many-on-many  test  scenarios. 

•  There  is  no  ctqiabUity  to  "fly"  the  test  article  in  a  complex  mission. 

Utility  of  the  Calverton  facility: 

•  The  chamber  has  been  used  to  support  tests  of  more  than  90  aircraft 

•  Its  low-frequency  performance  (60  MHz  to  500  MHz)  is  superior  to  ACETEF 
orBAF. 

•  Its  absorbent  cone  material  (horse  hair)  permits  higher  radiated  power  in  the 
chamber. 

•  It  is  big  enough  for  some  aircraft  (E-2C)  that  will  not  fit  into  ACETEF. 


G-10 


•  Where  test  users  do  not  require  complex  mission  scenarios  and  threat 
oivircnunents,  the  facility  is  available  and  may  provide  adequate  test  capability. 
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ABBREVIATIONS 


AATF 

Aircraft  Anechoic  Test  FaciliQr 

ACETEF 

Air  Combat  Environment  Test  and  Evaluation  Facility 

AF 

Air  Force 

AFB 

Air  Force  Base 

AFECO 

Air  Force  Electronic  Combat  Office 

AFFTC 

Air  Force  Flight  Test  Center 

ASEF 

Aircrew  Systems  Evaluation  Facility 

ASL 

Aircrew  Systems  Laboratory 

ATLAS 

Antenna  Test  Laboratory  Automated  System 

BAF 

Benefield  Anechoic  Facility 

CNIL 

communications,  navigation  and  identification 

CTEIP 

Central  Test  and  Evaluation  Investment  Program 

CTF 

Combined  Test  Force 

DoD 

Department  of  Defense 

DT&E 

development  test  and  evaluation 

E3 

electromagnetic  envirorunental  effects 

E3TL 

Electromagnetic  Enviroruner^al  Effects  Test  Laboratory 

EC 

electronic  combat 

ECTT 

Electronic  Combat  Integrated  Test  (AFFTC) 

EMEGS 

Electromagnetic  Environment  Generating  System 

BO 

electro-optical 

EW 

electronic  warfare 

EWISTL 

Electronic  Warfare  Integrated  Systems  Test  Laboratory 

FUR 

forward-looking  infrared 

FY 

fiscal  year 

FYDP 

Future  Years  Defense  Program 

GTRI 

Georgia  Tech  Research  Institute 

I&M 

Improvement  and  Modernization 

IDA 

Institute  for  Defense  Analyses 

DFAST 

Integration  Facility  for  Avionics  Systems  Test 

IFF 

identification  friend  or  foe 
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INATS 

Integrated  Avionics  Test  Station 

IR 

infrared 

IRST 

infrared  search  and  track 

ISTF 

Installed  System  Test  Facility 

LAC 

large  anechoic  chamber 

MFS 

manned  flight  simulator 

MILCON 

Military  Construcdon 

MRTFB 

Major  Range  and  Test  Facility  Base 

MTP 

Manage  to  Program 

NAS 

Naval  Air  Station 

NATC 

Naval  Air  Test  Center 

NAWC 

Naval  Air  Warfare  Center 

NAWC-AD 

Naval  Air  Warfare  Center- Aircraft  Division 

NIFFTE 

Navy  Identification  Friend  or  Foe  Test  and  Evaluation 

OCC 

Operations  and  Control  Center 

OSD 

Office  of  the  Secretary  of  Defense 

OSL 

Offensive  Sensors  Laboratory 

OT&E 

operational  test  and  evaluation 

P3l 

Pre-Planned  Product  Improvement 

PE 

program  element 

PMA 

program  management  activity 

POM 

Program  Objectives  Memorandum 

PRIMES 

Preflight  Integration  of  Munitions  and  Electronics  Systems 

R&M 

reliability  and  maintainability 

RCS 

radar  cross-section 

RDT&E 

research,  development,  test  and  evaluation 

RF 

radio  frequency 

RTS 

Radar  Test  System 

RUMS 

Range  Utilization  Measurement  System 

SAC 

small  anechoic  chamber 

SH 

shielded  hangar 

SIL 

systems  integration  laboratory 

SPO 

Systems  Program  Office 

SUT 

system  under  test 

SWEG 

Simulated  Warfare  Environment  Generator 

TASTEF 

Tactical  Avionics  Software  Test  and  Evaluation  Facility 
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TEMS 

TIS 


Test  and  Evaluation  Mission  Simulator 
test  information  sheet 
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